2025-04-25T04:42:34.8222593Z Current runner version: '2.323.0'
2025-04-25T04:42:34.8229193Z Runner name: 'gpu5966'
2025-04-25T04:42:34.8230369Z Machine name: 'gpu5966'
2025-04-25T04:42:34.8233540Z ##[group]GITHUB_TOKEN Permissions
2025-04-25T04:42:34.8235736Z Contents: read
2025-04-25T04:42:34.8236336Z Metadata: read
2025-04-25T04:42:34.8236985Z ##[endgroup]
2025-04-25T04:42:34.8239094Z Secret source: Actions
2025-04-25T04:42:34.8240016Z Prepare workflow directory
2025-04-25T04:42:35.5010577Z Prepare all required actions
2025-04-25T04:42:35.5055996Z Getting action download info
2025-04-25T04:42:35.7792518Z Download action repository 'pytorch/pytorch@main' (SHA:dda0c952e71a540f7ad8d040e35da727b4d91405)
2025-04-25T04:42:40.1268593Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502)
2025-04-25T04:42:40.6919092Z Download action repository 'aws-actions/amazon-ecr-login@v2' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076)
2025-04-25T04:42:41.1430010Z Download action repository 'pytorch/test-infra@main' (SHA:dba7ae1b796e6ea5268ffdec5594ba272b42a67a)
2025-04-25T04:42:41.9866072Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-04-25T04:42:42.6507025Z Getting action download info
2025-04-25T04:42:42.7767451Z Download action repository 'actions/checkout@v4' (SHA:11bd71901bbe5b1630ceea73d27597364c9af683)
2025-04-25T04:42:43.2942344Z Getting action download info
2025-04-25T04:42:43.4323522Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2025-04-25T04:42:43.8784353Z Getting action download info
2025-04-25T04:42:44.0083596Z Complete job name: linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T04:42:44.1731781Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2025-04-25T04:42:44.1732398Z with:
2025-04-25T04:42:44.1732574Z   no-sudo: true
2025-04-25T04:42:44.1732753Z   submodules: recursive
2025-04-25T04:42:44.1732935Z   fetch-depth: 0
2025-04-25T04:42:44.1733242Z env:
2025-04-25T04:42:44.1733399Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:42:44.1733579Z ##[endgroup]
2025-04-25T04:42:44.1803674Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-04-25T04:42:44.1804417Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-04-25T04:42:44.1834659Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:42:44.1834962Z env:
2025-04-25T04:42:44.1835132Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:42:44.1835324Z ##[endgroup]
2025-04-25T04:42:44.2170524Z ##[group]Run # Use all available CPUs for fetching
2025-04-25T04:42:44.2171356Z # Use all available CPUs for fetching
2025-04-25T04:42:44.2171971Z cd "${GITHUB_WORKSPACE}"
2025-04-25T04:42:44.2172603Z git config --global fetch.parallel 0
2025-04-25T04:42:44.2173287Z git config --global submodule.fetchJobs 0
2025-04-25T04:42:44.2173904Z 
2025-04-25T04:42:44.2174506Z # Clean workspace. The default checkout action should also do this, but
2025-04-25T04:42:44.2175317Z # do it here as well just in case
2025-04-25T04:42:44.2175880Z if [[ -d .git ]]; then
2025-04-25T04:42:44.2176382Z   if [ -z "${NO_SUDO}" ]; then
2025-04-25T04:42:44.2176926Z     sudo git clean -ffdx
2025-04-25T04:42:44.2177386Z   else
2025-04-25T04:42:44.2177800Z     git clean -ffdx
2025-04-25T04:42:44.2178231Z   fi
2025-04-25T04:42:44.2178593Z fi
2025-04-25T04:42:44.2231371Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:42:44.2231997Z env:
2025-04-25T04:42:44.2232327Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:42:44.2232704Z   NO_SUDO: true
2025-04-25T04:42:44.2233159Z ##[endgroup]
2025-04-25T04:42:44.6196143Z Removing .additional_ci_files/
2025-04-25T04:42:44.6197585Z Removing build/
2025-04-25T04:42:44.6197983Z Removing dist/
2025-04-25T04:42:44.6198410Z Removing test/test-reports/
2025-04-25T04:42:44.6327493Z ##[group]Run actions/checkout@v4
2025-04-25T04:42:44.6328017Z with:
2025-04-25T04:42:44.6328463Z   ref: b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:42:44.6329022Z   fetch-depth: 0
2025-04-25T04:42:44.6329434Z   submodules: recursive
2025-04-25T04:42:44.6329923Z   show-progress: false
2025-04-25T04:42:44.6330484Z   repository: pytorch/pytorch
2025-04-25T04:42:44.6331336Z   token: ***
2025-04-25T04:42:44.6331782Z   ssh-strict: true
2025-04-25T04:42:44.6332245Z   ssh-user: git
2025-04-25T04:42:44.6332710Z   persist-credentials: true
2025-04-25T04:42:44.6333245Z   clean: true
2025-04-25T04:42:44.6333741Z   sparse-checkout-cone-mode: true
2025-04-25T04:42:44.6334341Z   fetch-tags: false
2025-04-25T04:42:44.6334801Z   lfs: false
2025-04-25T04:42:44.6335245Z   set-safe-directory: true
2025-04-25T04:42:44.6335766Z env:
2025-04-25T04:42:44.6336169Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:42:44.6336626Z ##[endgroup]
2025-04-25T04:42:44.7731713Z Syncing repository: pytorch/pytorch
2025-04-25T04:42:44.7734264Z ##[group]Getting Git version info
2025-04-25T04:42:44.7735192Z Working directory is '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch'
2025-04-25T04:42:44.7736584Z [command]/usr/bin/git version
2025-04-25T04:42:44.7737190Z git version 2.34.1
2025-04-25T04:42:44.7738928Z ##[endgroup]
2025-04-25T04:42:44.7745755Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/814a2806-d6f7-400f-bf30-9dace0d70a62/.gitconfig'
2025-04-25T04:42:44.7748547Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/814a2806-d6f7-400f-bf30-9dace0d70a62' before making global git config changes
2025-04-25T04:42:44.7750238Z Adding repository directory to the temporary git global config as a safe directory
2025-04-25T04:42:44.7751605Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-04-25T04:42:44.7807667Z [command]/usr/bin/git config --local --get remote.origin.url
2025-04-25T04:42:44.7857571Z https://github.com/pytorch/pytorch
2025-04-25T04:42:44.7874970Z ##[group]Removing previously created refs, to avoid conflicts
2025-04-25T04:42:44.7876421Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
2025-04-25T04:42:44.7926566Z HEAD
2025-04-25T04:42:44.8007253Z ##[endgroup]
2025-04-25T04:42:44.8011997Z [command]/usr/bin/git submodule status
2025-04-25T04:42:44.8798545Z  7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe)
2025-04-25T04:42:44.9023454Z  4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081)
2025-04-25T04:42:44.9250794Z  b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327)
2025-04-25T04:42:44.9500183Z  c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0)
2025-04-25T04:42:44.9565092Z  e170594ac7cf1dac584da473d4ca9301087090c1 third_party/NVTX (v3.1.0)
2025-04-25T04:42:44.9736301Z  a6bfc237255a6bac1513f7c1ebde6d8aed6b5191 third_party/VulkanMemoryAllocator (v2.1.0-705-ga6bfc23)
2025-04-25T04:42:45.0643057Z  51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a010365)
2025-04-25T04:42:45.0712696Z  0d98dba29d66e93259db7daa53a9327df767a415 third_party/benchmark (v1.6.1)
2025-04-25T04:42:45.0795826Z  8086bbe3a78d931eb96fe12fdc014082e18d18d3 third_party/composable_kernel (mock-tag-test-6-g8086bbe3a)
2025-04-25T04:42:45.1108956Z  3b6597bba913d51161383657829b7e644e59c006 third_party/cpp-httplib (v0.15.3-20-g3b6597b)
2025-04-25T04:42:45.1423843Z  5e3d2445e6a84d9599bee2bf78edbb4d80865e1d third_party/cpuinfo (5e3d244)
2025-04-25T04:42:45.1517758Z  8801fd7b31c2f798732ed1e1dd4711a9cde4217d third_party/cudnn_frontend (v0.5-46-g8801fd7)
2025-04-25T04:42:45.1739883Z  afa1772203677c5118fcd82537a9c8fefbcc7008 third_party/cutlass (v3.8.0)
2025-04-25T04:42:45.2649562Z  3147391d946bb4b6c68edd901f2add6ac1f31f8c third_party/eigen (3.4.0)
2025-04-25T04:42:45.3381600Z  dbc3157bf256f1339b3fa1fef2be89ac4078be0e third_party/fbgemm (v0.4.1-446-gdbc3157b)
2025-04-25T04:42:45.3594575Z  979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4)
2025-04-25T04:42:45.3661670Z  01834de25e4bf3975a9a00e816292b1ad0fe184b third_party/flatbuffers (v23.3.3)
2025-04-25T04:42:45.4355413Z  123913715afeb8a437e6388b4473fcc4753e1c9a third_party/fmt (11.1.4)
2025-04-25T04:42:45.4639443Z  3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17)
2025-04-25T04:42:45.4865462Z  c61070427610ccd923efe3e7f8b3eca12bbcc31a third_party/gloo (remotes/origin/HEAD)
2025-04-25T04:42:45.5338441Z  b514bdc898e2951020cbdca1304b75f5950d1f59 third_party/googletest (release-1.8.0-3484-gb514bdc8)
2025-04-25T04:42:45.5544789Z  719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1)
2025-04-25T04:42:45.5703167Z  5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42 third_party/ittapi (v3.23.0-14-g5b8a7d7)
2025-04-25T04:42:45.6318577Z  d82680bbd44f872aa04394fa5bba23a7992f9fa4 third_party/kineto (heads/main-8-gd82680b)
2025-04-25T04:42:45.6387647Z  ef685a13cfbe8d418aa2ed34350e21e4938358b6 third_party/kleidiai (v1.3.0)
2025-04-25T04:42:45.6456145Z  b66e3214d8a104669c2ec05ae91ebc26a8f5ab78 third_party/mimalloc (v1.8.2)
2025-04-25T04:42:45.7330632Z  87cda1d6646592ac5866dc703c8e1839046a6806 third_party/nlohmann (v3.10.1-113-g87cda1d6)
2025-04-25T04:42:45.7897699Z  b8baa8446686496da4cc8fda09f2b6fe65c2a02c third_party/onnx (v1.17.0)
2025-04-25T04:42:45.7967579Z  a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2)
2025-04-25T04:42:45.8040539Z  9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe third_party/pocketfft (release_for_eigen-11-g9d3ab05)
2025-04-25T04:42:45.8713844Z  d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b)
2025-04-25T04:42:45.8932906Z  072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master)
2025-04-25T04:42:45.9084878Z  4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e)
2025-04-25T04:42:45.9211406Z  a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e)
2025-04-25T04:42:45.9434324Z  f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated)
2025-04-25T04:42:45.9639416Z  5a1d179df9cf652951b59010a2d2075372d67f68 third_party/sleef (3.8)
2025-04-25T04:42:45.9847302Z  52791a2fd214b2a9dc5759d36725909c1daa7f2e third_party/tensorpipe (remotes/origin/master)
2025-04-25T04:42:45.9879137Z ##[group]Cleaning the repository
2025-04-25T04:42:45.9886657Z [command]/usr/bin/git clean -ffdx
2025-04-25T04:42:46.0476541Z [command]/usr/bin/git reset --hard HEAD
2025-04-25T04:42:46.2229804Z HEAD is now at 66b62988318 no sym_or
2025-04-25T04:42:46.2275654Z ##[endgroup]
2025-04-25T04:42:46.2277025Z ##[group]Disabling automatic garbage collection
2025-04-25T04:42:46.2287406Z [command]/usr/bin/git config --local gc.auto 0
2025-04-25T04:42:46.2361010Z ##[endgroup]
2025-04-25T04:42:46.2361799Z ##[group]Setting up auth
2025-04-25T04:42:46.2374181Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-04-25T04:42:46.2471015Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-04-25T04:42:46.3165908Z Entering 'android/libs/fbjni'
2025-04-25T04:42:46.3289132Z Entering 'third_party/FP16'
2025-04-25T04:42:46.3418592Z Entering 'third_party/FXdiv'
2025-04-25T04:42:46.3544637Z Entering 'third_party/NNPACK'
2025-04-25T04:42:46.3666767Z Entering 'third_party/NVTX'
2025-04-25T04:42:46.3792055Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-25T04:42:46.3914032Z Entering 'third_party/XNNPACK'
2025-04-25T04:42:46.4070246Z Entering 'third_party/benchmark'
2025-04-25T04:42:46.4191205Z Entering 'third_party/composable_kernel'
2025-04-25T04:42:46.4329893Z Entering 'third_party/cpp-httplib'
2025-04-25T04:42:46.4451258Z Entering 'third_party/cpuinfo'
2025-04-25T04:42:46.4572822Z Entering 'third_party/cudnn_frontend'
2025-04-25T04:42:46.4691120Z Entering 'third_party/cutlass'
2025-04-25T04:42:46.4827574Z Entering 'third_party/eigen'
2025-04-25T04:42:46.4950644Z Entering 'third_party/fbgemm'
2025-04-25T04:42:46.5069150Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-25T04:42:46.5184231Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T04:42:46.5300009Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-25T04:42:46.5428123Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-25T04:42:46.5542620Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T04:42:46.5662803Z Entering 'third_party/flash-attention'
2025-04-25T04:42:46.5784124Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T04:42:46.5910822Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-25T04:42:46.6056330Z Entering 'third_party/flatbuffers'
2025-04-25T04:42:46.6183023Z Entering 'third_party/fmt'
2025-04-25T04:42:46.6301566Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-25T04:42:46.6421269Z Entering 'third_party/gloo'
2025-04-25T04:42:46.6541016Z Entering 'third_party/googletest'
2025-04-25T04:42:46.6661593Z Entering 'third_party/ideep'
2025-04-25T04:42:46.6775231Z Entering 'third_party/ideep/mkl-dnn'
2025-04-25T04:42:46.6911368Z Entering 'third_party/ittapi'
2025-04-25T04:42:46.7031148Z Entering 'third_party/kineto'
2025-04-25T04:42:46.7147231Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T04:42:46.7257780Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T04:42:46.7374079Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T04:42:46.7483865Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T04:42:46.7594937Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T04:42:46.7702588Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T04:42:46.7823573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T04:42:46.7936772Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T04:42:46.8049824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T04:42:46.8164735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T04:42:46.8284450Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T04:42:46.8396816Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T04:42:46.8519096Z Entering 'third_party/kleidiai'
2025-04-25T04:42:46.8643404Z Entering 'third_party/mimalloc'
2025-04-25T04:42:46.8767488Z Entering 'third_party/nlohmann'
2025-04-25T04:42:46.8892784Z Entering 'third_party/onnx'
2025-04-25T04:42:46.9044225Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-25T04:42:46.9171978Z Entering 'third_party/opentelemetry-cpp'
2025-04-25T04:42:46.9294985Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T04:42:46.9408146Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T04:42:46.9520623Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T04:42:46.9637753Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T04:42:46.9755848Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T04:42:46.9871591Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T04:42:46.9987944Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T04:42:47.0100228Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T04:42:47.0218769Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T04:42:47.0342017Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T04:42:47.0503523Z Entering 'third_party/pocketfft'
2025-04-25T04:42:47.0623241Z Entering 'third_party/protobuf'
2025-04-25T04:42:47.0746712Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-25T04:42:47.0860273Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-25T04:42:47.0987935Z Entering 'third_party/psimd'
2025-04-25T04:42:47.1110211Z Entering 'third_party/pthreadpool'
2025-04-25T04:42:47.1234205Z Entering 'third_party/pybind11'
2025-04-25T04:42:47.1359710Z Entering 'third_party/python-peachpy'
2025-04-25T04:42:47.1486531Z Entering 'third_party/sleef'
2025-04-25T04:42:47.1607144Z Entering 'third_party/tensorpipe'
2025-04-25T04:42:47.1728912Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-25T04:42:47.1850772Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-25T04:42:47.1967867Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-25T04:42:47.2089720Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T04:42:47.2200561Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T04:42:47.2372581Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-04-25T04:42:47.2444521Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-04-25T04:42:47.3112125Z Entering 'android/libs/fbjni'
2025-04-25T04:42:47.3234061Z Entering 'third_party/FP16'
2025-04-25T04:42:47.3353809Z Entering 'third_party/FXdiv'
2025-04-25T04:42:47.3473367Z Entering 'third_party/NNPACK'
2025-04-25T04:42:47.3591628Z Entering 'third_party/NVTX'
2025-04-25T04:42:47.3715526Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-25T04:42:47.3834983Z Entering 'third_party/XNNPACK'
2025-04-25T04:42:47.3988012Z Entering 'third_party/benchmark'
2025-04-25T04:42:47.4107271Z Entering 'third_party/composable_kernel'
2025-04-25T04:42:47.4240734Z Entering 'third_party/cpp-httplib'
2025-04-25T04:42:47.4359931Z Entering 'third_party/cpuinfo'
2025-04-25T04:42:47.4477746Z Entering 'third_party/cudnn_frontend'
2025-04-25T04:42:47.4597073Z Entering 'third_party/cutlass'
2025-04-25T04:42:47.4735820Z Entering 'third_party/eigen'
2025-04-25T04:42:47.4859575Z Entering 'third_party/fbgemm'
2025-04-25T04:42:47.4976969Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-25T04:42:47.5087527Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T04:42:47.5198746Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-25T04:42:47.5328280Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-25T04:42:47.5439869Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T04:42:47.5561352Z Entering 'third_party/flash-attention'
2025-04-25T04:42:47.5679021Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T04:42:47.5806800Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-25T04:42:47.5947115Z Entering 'third_party/flatbuffers'
2025-04-25T04:42:47.6073249Z Entering 'third_party/fmt'
2025-04-25T04:42:47.6194402Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-25T04:42:47.6316455Z Entering 'third_party/gloo'
2025-04-25T04:42:47.6437296Z Entering 'third_party/googletest'
2025-04-25T04:42:47.6557919Z Entering 'third_party/ideep'
2025-04-25T04:42:47.6675642Z Entering 'third_party/ideep/mkl-dnn'
2025-04-25T04:42:47.6811420Z Entering 'third_party/ittapi'
2025-04-25T04:42:47.6933174Z Entering 'third_party/kineto'
2025-04-25T04:42:47.7051876Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T04:42:47.7162773Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T04:42:47.7279833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T04:42:47.7392737Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T04:42:47.7502492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T04:42:47.7610543Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T04:42:47.7729179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T04:42:47.7839911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T04:42:47.7953753Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T04:42:47.8067690Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T04:42:47.8188355Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T04:42:47.8300290Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T04:42:47.8419237Z Entering 'third_party/kleidiai'
2025-04-25T04:42:47.8538263Z Entering 'third_party/mimalloc'
2025-04-25T04:42:47.8657565Z Entering 'third_party/nlohmann'
2025-04-25T04:42:47.8778771Z Entering 'third_party/onnx'
2025-04-25T04:42:47.8926302Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-25T04:42:47.9051344Z Entering 'third_party/opentelemetry-cpp'
2025-04-25T04:42:47.9169720Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T04:42:47.9278427Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T04:42:47.9390247Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T04:42:47.9498852Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T04:42:47.9611588Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T04:42:47.9722212Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T04:42:47.9836104Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T04:42:47.9941931Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T04:42:48.0058704Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T04:42:48.0174993Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T04:42:48.0330601Z Entering 'third_party/pocketfft'
2025-04-25T04:42:48.0448041Z Entering 'third_party/protobuf'
2025-04-25T04:42:48.0572174Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-25T04:42:48.0682651Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-25T04:42:48.0799983Z Entering 'third_party/psimd'
2025-04-25T04:42:48.0913931Z Entering 'third_party/pthreadpool'
2025-04-25T04:42:48.1031822Z Entering 'third_party/pybind11'
2025-04-25T04:42:48.1150899Z Entering 'third_party/python-peachpy'
2025-04-25T04:42:48.1269855Z Entering 'third_party/sleef'
2025-04-25T04:42:48.1388852Z Entering 'third_party/tensorpipe'
2025-04-25T04:42:48.1506713Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-25T04:42:48.1621980Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-25T04:42:48.1734362Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-25T04:42:48.1850603Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T04:42:48.1965736Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T04:42:48.2137927Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-04-25T04:42:48.2230189Z ##[endgroup]
2025-04-25T04:42:48.2231033Z ##[group]Fetching the repository
2025-04-25T04:42:48.2245389Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2025-04-25T04:42:48.8847837Z From https://github.com/pytorch/pytorch
2025-04-25T04:42:48.8848729Z  - [deleted]                 (none)     -> ciflow/inductor/152149
2025-04-25T04:42:48.9512023Z  - [deleted]                 (none)     -> ciflow/rocm-mi300/148945
2025-04-25T04:42:48.9512868Z  - [deleted]                 (none)     -> ciflow/rocm/148945
2025-04-25T04:42:48.9513629Z  - [deleted]                 (none)     -> ciflow/trunk/150784
2025-04-25T04:42:48.9514376Z  - [deleted]                 (none)     -> ciflow/trunk/151897
2025-04-25T04:42:48.9515745Z  - [deleted]                 (none)     -> ciflow/trunk/152149
2025-04-25T04:42:50.6874514Z    e9a92898cf8..6b6427a16d6  divup                       -> origin/divup
2025-04-25T04:42:50.6935221Z    de5b0ece3da..182033fffc2  gh/dharakk/2/head           -> origin/gh/dharakk/2/head
2025-04-25T04:42:50.6940425Z  + 7b17b83ef01...7bdd185cc58 gh/dharakk/2/orig           -> origin/gh/dharakk/2/orig  (forced update)
2025-04-25T04:42:50.7247807Z    6d02a0b9f90..cdd1653b69a  gh/malfet/197/base          -> origin/gh/malfet/197/base
2025-04-25T04:42:50.7254867Z    ccd7eae1024..914692de647  gh/malfet/197/head          -> origin/gh/malfet/197/head
2025-04-25T04:42:50.7257341Z  + bfc6fb99488...4bc7efa826a gh/malfet/197/orig          -> origin/gh/malfet/197/orig  (forced update)
2025-04-25T04:42:50.7292875Z  * [new branch]              gh/soulitzer/358/base       -> origin/gh/soulitzer/358/base
2025-04-25T04:42:50.7293962Z  * [new branch]              gh/soulitzer/358/head       -> origin/gh/soulitzer/358/head
2025-04-25T04:42:50.7295017Z  * [new branch]              gh/soulitzer/358/orig       -> origin/gh/soulitzer/358/orig
2025-04-25T04:42:50.7297703Z  * [new branch]              gh/soulitzer/359/base       -> origin/gh/soulitzer/359/base
2025-04-25T04:42:50.7300806Z  * [new branch]              gh/soulitzer/359/head       -> origin/gh/soulitzer/359/head
2025-04-25T04:42:50.7303967Z  * [new branch]              gh/soulitzer/359/orig       -> origin/gh/soulitzer/359/orig
2025-04-25T04:42:50.7335199Z    1a6d50d4079..dda0c952e71  main                        -> origin/main
2025-04-25T04:42:50.7347790Z    66b62988318..c44e78b0e5f  pianpwk/false_infer_size    -> origin/pianpwk/false_infer_size
2025-04-25T04:42:50.7358545Z    fc6e37ceb23..75c71ab3714  viable/strict               -> origin/viable/strict
2025-04-25T04:42:50.7365054Z  * [new tag]                 ciflow/inductor-rocm/151708 -> ciflow/inductor-rocm/151708
2025-04-25T04:42:50.7386773Z  t [tag update]              ciflow/inductor/151990      -> ciflow/inductor/151990
2025-04-25T04:42:50.7391123Z  t [tag update]              ciflow/inductor/152146      -> ciflow/inductor/152146
2025-04-25T04:42:50.7392861Z  * [new tag]                 ciflow/inductor/152159      -> ciflow/inductor/152159
2025-04-25T04:42:50.7397807Z  t [tag update]              ciflow/mps/147893           -> ciflow/mps/147893
2025-04-25T04:42:50.7400447Z  * [new tag]                 ciflow/periodic-rocm-mi300/151708 -> ciflow/periodic-rocm-mi300/151708
2025-04-25T04:42:50.7403886Z  * [new tag]                 ciflow/rocm-mi300/151708    -> ciflow/rocm-mi300/151708
2025-04-25T04:42:50.7409083Z  t [tag update]              ciflow/rocm/152133          -> ciflow/rocm/152133
2025-04-25T04:42:50.7414203Z  * [new tag]                 ciflow/trunk/148893         -> ciflow/trunk/148893
2025-04-25T04:42:50.7419605Z  t [tag update]              ciflow/trunk/151974         -> ciflow/trunk/151974
2025-04-25T04:42:50.7420491Z  * [new tag]                 ciflow/trunk/152071         -> ciflow/trunk/152071
2025-04-25T04:42:50.7421796Z  * [new tag]                 ciflow/trunk/152111         -> ciflow/trunk/152111
2025-04-25T04:42:50.7423198Z  t [tag update]              ciflow/trunk/152133         -> ciflow/trunk/152133
2025-04-25T04:42:50.8399938Z [command]/usr/bin/git rev-parse --verify --quiet b68c0effe97351b0c24b2f7aef671559be6aa977^{object}
2025-04-25T04:42:50.8516154Z b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:42:50.8530105Z ##[endgroup]
2025-04-25T04:42:50.8530836Z ##[group]Determining the checkout info
2025-04-25T04:42:50.8531924Z ##[endgroup]
2025-04-25T04:42:50.8540175Z [command]/usr/bin/git sparse-checkout disable
2025-04-25T04:42:50.8895617Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-04-25T04:42:50.8959793Z ##[group]Checking out the ref
2025-04-25T04:42:50.8966415Z [command]/usr/bin/git checkout --progress --force b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:42:51.2176578Z Previous HEAD position was 66b62988318 no sym_or
2025-04-25T04:42:51.2194789Z HEAD is now at b68c0effe97 Update on "use statically known true instead of guard size oblivious in bmm and mm decompositions . "
2025-04-25T04:42:51.2302616Z ##[endgroup]
2025-04-25T04:42:51.2303461Z ##[group]Setting up auth for fetching submodules
2025-04-25T04:42:51.2313861Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-04-25T04:42:51.2404537Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2025-04-25T04:42:51.2482796Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2025-04-25T04:42:51.2562515Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2025-04-25T04:42:51.2635978Z ##[endgroup]
2025-04-25T04:42:51.2636693Z ##[group]Fetching submodules
2025-04-25T04:42:51.2643944Z [command]/usr/bin/git submodule sync --recursive
2025-04-25T04:42:51.3331289Z Synchronizing submodule url for 'android/libs/fbjni'
2025-04-25T04:42:51.3434212Z Synchronizing submodule url for 'third_party/FP16'
2025-04-25T04:42:51.3544505Z Synchronizing submodule url for 'third_party/FXdiv'
2025-04-25T04:42:51.3648360Z Synchronizing submodule url for 'third_party/NNPACK'
2025-04-25T04:42:51.3755319Z Synchronizing submodule url for 'third_party/NVTX'
2025-04-25T04:42:51.3859561Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator'
2025-04-25T04:42:51.3964082Z Synchronizing submodule url for 'third_party/XNNPACK'
2025-04-25T04:42:51.4100723Z Synchronizing submodule url for 'third_party/benchmark'
2025-04-25T04:42:51.4204390Z Synchronizing submodule url for 'third_party/composable_kernel'
2025-04-25T04:42:51.4326497Z Synchronizing submodule url for 'third_party/cpp-httplib'
2025-04-25T04:42:51.4424477Z Synchronizing submodule url for 'third_party/cpuinfo'
2025-04-25T04:42:51.4529110Z Synchronizing submodule url for 'third_party/cudnn_frontend'
2025-04-25T04:42:51.4632358Z Synchronizing submodule url for 'third_party/cutlass'
2025-04-25T04:42:51.4754912Z Synchronizing submodule url for 'third_party/eigen'
2025-04-25T04:42:51.4863111Z Synchronizing submodule url for 'third_party/fbgemm'
2025-04-25T04:42:51.4960129Z Synchronizing submodule url for 'third_party/fbgemm/third_party/asmjit'
2025-04-25T04:42:51.5059412Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T04:42:51.5153085Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cutlass'
2025-04-25T04:42:51.5266757Z Synchronizing submodule url for 'third_party/fbgemm/third_party/googletest'
2025-04-25T04:42:51.5367962Z Synchronizing submodule url for 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T04:42:51.5472583Z Synchronizing submodule url for 'third_party/flash-attention'
2025-04-25T04:42:51.5574389Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T04:42:51.5694061Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass'
2025-04-25T04:42:51.5820993Z Synchronizing submodule url for 'third_party/flatbuffers'
2025-04-25T04:42:51.5931125Z Synchronizing submodule url for 'third_party/fmt'
2025-04-25T04:42:51.6031577Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp'
2025-04-25T04:42:51.6135696Z Synchronizing submodule url for 'third_party/gloo'
2025-04-25T04:42:51.6240166Z Synchronizing submodule url for 'third_party/googletest'
2025-04-25T04:42:51.6346401Z Synchronizing submodule url for 'third_party/ideep'
2025-04-25T04:42:51.6444606Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn'
2025-04-25T04:42:51.6566557Z Synchronizing submodule url for 'third_party/ittapi'
2025-04-25T04:42:51.6674649Z Synchronizing submodule url for 'third_party/kineto'
2025-04-25T04:42:51.6775305Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T04:42:51.6875210Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T04:42:51.6977985Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T04:42:51.7080145Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T04:42:51.7179048Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T04:42:51.7267178Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T04:42:51.7371885Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T04:42:51.7467593Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T04:42:51.7565619Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T04:42:51.7660412Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T04:42:51.7763880Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T04:42:51.7858257Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T04:42:51.7964730Z Synchronizing submodule url for 'third_party/kleidiai'
2025-04-25T04:42:51.8071753Z Synchronizing submodule url for 'third_party/mimalloc'
2025-04-25T04:42:51.8177371Z Synchronizing submodule url for 'third_party/nlohmann'
2025-04-25T04:42:51.8283453Z Synchronizing submodule url for 'third_party/onnx'
2025-04-25T04:42:51.8416088Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11'
2025-04-25T04:42:51.8531940Z Synchronizing submodule url for 'third_party/opentelemetry-cpp'
2025-04-25T04:42:51.8632773Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T04:42:51.8729153Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T04:42:51.8825465Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T04:42:51.8923769Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T04:42:51.9021141Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T04:42:51.9115898Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T04:42:51.9205254Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T04:42:51.9290449Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T04:42:51.9393475Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T04:42:51.9501466Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T04:42:51.9645803Z Synchronizing submodule url for 'third_party/pocketfft'
2025-04-25T04:42:51.9752091Z Synchronizing submodule url for 'third_party/protobuf'
2025-04-25T04:42:51.9858465Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark'
2025-04-25T04:42:51.9956386Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest'
2025-04-25T04:42:52.0069785Z Synchronizing submodule url for 'third_party/psimd'
2025-04-25T04:42:52.0178179Z Synchronizing submodule url for 'third_party/pthreadpool'
2025-04-25T04:42:52.0284749Z Synchronizing submodule url for 'third_party/pybind11'
2025-04-25T04:42:52.0389175Z Synchronizing submodule url for 'third_party/python-peachpy'
2025-04-25T04:42:52.0491499Z Synchronizing submodule url for 'third_party/sleef'
2025-04-25T04:42:52.0599400Z Synchronizing submodule url for 'third_party/tensorpipe'
2025-04-25T04:42:52.0701888Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest'
2025-04-25T04:42:52.0794027Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop'
2025-04-25T04:42:52.0886129Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv'
2025-04-25T04:42:52.0984123Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T04:42:52.1070720Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T04:42:52.1213386Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2025-04-25T04:42:52.2475500Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2025-04-25T04:42:52.3093361Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2025-04-25T04:42:52.3710177Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2025-04-25T04:42:52.4350142Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2025-04-25T04:42:52.5004275Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1'
2025-04-25T04:42:52.5652990Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2025-04-25T04:42:52.6660415Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883'
2025-04-25T04:42:52.7366842Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2025-04-25T04:42:52.8351358Z Submodule path 'third_party/composable_kernel': checked out '8086bbe3a78d931eb96fe12fdc014082e18d18d3'
2025-04-25T04:42:52.9057862Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2025-04-25T04:42:52.9706909Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d'
2025-04-25T04:42:53.0397065Z Submodule path 'third_party/cudnn_frontend': checked out '8801fd7b31c2f798732ed1e1dd4711a9cde4217d'
2025-04-25T04:42:53.1216550Z Submodule path 'third_party/cutlass': checked out 'afa1772203677c5118fcd82537a9c8fefbcc7008'
2025-04-25T04:42:53.1993794Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2025-04-25T04:42:53.2764950Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2025-04-25T04:42:53.3317535Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2025-04-25T04:42:53.3925566Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2025-04-25T04:42:53.4736071Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2025-04-25T04:42:53.5320452Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2025-04-25T04:42:53.5898496Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2025-04-25T04:42:53.6624471Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5'
2025-04-25T04:42:53.7529076Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33'
2025-04-25T04:42:53.8331926Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420'
2025-04-25T04:42:53.9101844Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2025-04-25T04:42:53.9773678Z Submodule path 'third_party/fmt': checked out '123913715afeb8a437e6388b4473fcc4753e1c9a'
2025-04-25T04:42:54.0408663Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2025-04-25T04:42:54.1069779Z Submodule path 'third_party/gloo': checked out 'c61070427610ccd923efe3e7f8b3eca12bbcc31a'
2025-04-25T04:42:54.1719371Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59'
2025-04-25T04:42:54.2370997Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3'
2025-04-25T04:42:54.3304820Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d'
2025-04-25T04:42:54.3984369Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2025-04-25T04:42:54.4738225Z Submodule path 'third_party/kineto': checked out '5fa4bd8c8fb581a621f72957d0cb11431cc4714f'
2025-04-25T04:42:54.5389783Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2025-04-25T04:42:54.6024813Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2025-04-25T04:42:54.6648645Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2025-04-25T04:42:54.7268554Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2025-04-25T04:42:54.7862253Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2025-04-25T04:42:54.8436372Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2025-04-25T04:42:54.9052452Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2025-04-25T04:42:54.9664029Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2025-04-25T04:42:55.0348414Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2025-04-25T04:42:55.0930583Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2025-04-25T04:42:55.1547603Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2025-04-25T04:42:55.2146874Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2025-04-25T04:42:55.2825891Z Submodule path 'third_party/kleidiai': checked out 'ef685a13cfbe8d418aa2ed34350e21e4938358b6'
2025-04-25T04:42:55.3523527Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2025-04-25T04:42:55.4266422Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2025-04-25T04:42:55.5256426Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c'
2025-04-25T04:42:55.5986349Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b'
2025-04-25T04:42:55.6772554Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2025-04-25T04:42:55.7364625Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2025-04-25T04:42:55.7967606Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2025-04-25T04:42:55.8565790Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2025-04-25T04:42:55.9247930Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2025-04-25T04:42:55.9845976Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2025-04-25T04:42:56.0419060Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2025-04-25T04:42:56.1031216Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2025-04-25T04:42:56.1668435Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2025-04-25T04:42:56.2280732Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-04-25T04:42:56.3249527Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2025-04-25T04:42:56.3967781Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2025-04-25T04:42:56.4992911Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2025-04-25T04:42:56.5593007Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2025-04-25T04:42:56.6206187Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2025-04-25T04:42:56.6834515Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2025-04-25T04:42:56.7448940Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2025-04-25T04:42:56.8137249Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-04-25T04:42:56.8756991Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2025-04-25T04:42:56.9503772Z Submodule path 'third_party/sleef': checked out '56e1f79cb140fb9326d612d0be06b5250565cade'
2025-04-25T04:42:57.0163443Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2025-04-25T04:42:57.0736504Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2025-04-25T04:42:57.1306027Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2025-04-25T04:42:57.2217715Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2025-04-25T04:42:57.2854623Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2025-04-25T04:42:57.3404411Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2025-04-25T04:42:57.3647859Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2025-04-25T04:42:57.4316104Z Entering 'android/libs/fbjni'
2025-04-25T04:42:57.4424455Z Entering 'third_party/FP16'
2025-04-25T04:42:57.4538324Z Entering 'third_party/FXdiv'
2025-04-25T04:42:57.4646332Z Entering 'third_party/NNPACK'
2025-04-25T04:42:57.4755550Z Entering 'third_party/NVTX'
2025-04-25T04:42:57.4864359Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-25T04:42:57.4972464Z Entering 'third_party/XNNPACK'
2025-04-25T04:42:57.5119029Z Entering 'third_party/benchmark'
2025-04-25T04:42:57.5228586Z Entering 'third_party/composable_kernel'
2025-04-25T04:42:57.5348795Z Entering 'third_party/cpp-httplib'
2025-04-25T04:42:57.5455888Z Entering 'third_party/cpuinfo'
2025-04-25T04:42:57.5562471Z Entering 'third_party/cudnn_frontend'
2025-04-25T04:42:57.5669258Z Entering 'third_party/cutlass'
2025-04-25T04:42:57.5794236Z Entering 'third_party/eigen'
2025-04-25T04:42:57.5907191Z Entering 'third_party/fbgemm'
2025-04-25T04:42:57.6012514Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-25T04:42:57.6112075Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T04:42:57.6213110Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-25T04:42:57.6330680Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-25T04:42:57.6427731Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T04:42:57.6531046Z Entering 'third_party/flash-attention'
2025-04-25T04:42:57.6635874Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T04:42:57.6747484Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-25T04:42:57.6869597Z Entering 'third_party/flatbuffers'
2025-04-25T04:42:57.6980291Z Entering 'third_party/fmt'
2025-04-25T04:42:57.7088645Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-25T04:42:57.7193797Z Entering 'third_party/gloo'
2025-04-25T04:42:57.7298329Z Entering 'third_party/googletest'
2025-04-25T04:42:57.7404988Z Entering 'third_party/ideep'
2025-04-25T04:42:57.7503159Z Entering 'third_party/ideep/mkl-dnn'
2025-04-25T04:42:57.7621163Z Entering 'third_party/ittapi'
2025-04-25T04:42:57.7725643Z Entering 'third_party/kineto'
2025-04-25T04:42:57.7828950Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T04:42:57.7927262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T04:42:57.8028671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T04:42:57.8133578Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T04:42:57.8230352Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T04:42:57.8324539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T04:42:57.8429866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T04:42:57.8526997Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T04:42:57.8624672Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T04:42:57.8723295Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T04:42:57.8827194Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T04:42:57.8925209Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T04:42:57.9033696Z Entering 'third_party/kleidiai'
2025-04-25T04:42:57.9140105Z Entering 'third_party/mimalloc'
2025-04-25T04:42:57.9249902Z Entering 'third_party/nlohmann'
2025-04-25T04:42:57.9360304Z Entering 'third_party/onnx'
2025-04-25T04:42:57.9496689Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-25T04:42:57.9610002Z Entering 'third_party/opentelemetry-cpp'
2025-04-25T04:42:57.9717218Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T04:42:57.9817585Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T04:42:57.9915471Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T04:42:58.0014281Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T04:42:58.0116532Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T04:42:58.0212878Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T04:42:58.0310801Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T04:42:58.0403493Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T04:42:58.0505753Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T04:42:58.0613510Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T04:42:58.0759795Z Entering 'third_party/pocketfft'
2025-04-25T04:42:58.0864673Z Entering 'third_party/protobuf'
2025-04-25T04:42:58.0977775Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-25T04:42:58.1082712Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-25T04:42:58.1192287Z Entering 'third_party/psimd'
2025-04-25T04:42:58.1299428Z Entering 'third_party/pthreadpool'
2025-04-25T04:42:58.1408828Z Entering 'third_party/pybind11'
2025-04-25T04:42:58.1516805Z Entering 'third_party/python-peachpy'
2025-04-25T04:42:58.1624464Z Entering 'third_party/sleef'
2025-04-25T04:42:58.1730817Z Entering 'third_party/tensorpipe'
2025-04-25T04:42:58.1833320Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-25T04:42:58.1934390Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-25T04:42:58.2032207Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-25T04:42:58.2137194Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T04:42:58.2232719Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T04:42:58.2379219Z ##[endgroup]
2025-04-25T04:42:58.2380055Z ##[group]Persisting credentials for submodules
2025-04-25T04:42:58.2391947Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2025-04-25T04:42:58.3063729Z Entering 'android/libs/fbjni'
2025-04-25T04:42:58.3129462Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3130225Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3223416Z Entering 'third_party/FP16'
2025-04-25T04:42:58.3288608Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3289328Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3386266Z Entering 'third_party/FXdiv'
2025-04-25T04:42:58.3451204Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3451973Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3542440Z Entering 'third_party/NNPACK'
2025-04-25T04:42:58.3605667Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3606403Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3698851Z Entering 'third_party/NVTX'
2025-04-25T04:42:58.3763310Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3763994Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3858203Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-25T04:42:58.3920188Z url.https://github.com/.insteadof
2025-04-25T04:42:58.3920846Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4012325Z Entering 'third_party/XNNPACK'
2025-04-25T04:42:58.4075227Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4075916Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4203410Z Entering 'third_party/benchmark'
2025-04-25T04:42:58.4266980Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4267655Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4357313Z Entering 'third_party/composable_kernel'
2025-04-25T04:42:58.4421889Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4422633Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4527921Z Entering 'third_party/cpp-httplib'
2025-04-25T04:42:58.4588621Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4589306Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4681076Z Entering 'third_party/cpuinfo'
2025-04-25T04:42:58.4744912Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4745610Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4836577Z Entering 'third_party/cudnn_frontend'
2025-04-25T04:42:58.4900494Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4901166Z url.https://github.com/.insteadof
2025-04-25T04:42:58.4993116Z Entering 'third_party/cutlass'
2025-04-25T04:42:58.5057079Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5057836Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5169576Z Entering 'third_party/eigen'
2025-04-25T04:42:58.5231933Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5232684Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5328957Z Entering 'third_party/fbgemm'
2025-04-25T04:42:58.5392502Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5393266Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5484241Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-25T04:42:58.5545232Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5545947Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5635472Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T04:42:58.5696926Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5697598Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5792348Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-25T04:42:58.5852695Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5853434Z url.https://github.com/.insteadof
2025-04-25T04:42:58.5958293Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-25T04:42:58.6018912Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6019587Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6108652Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T04:42:58.6167695Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6168421Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6265741Z Entering 'third_party/flash-attention'
2025-04-25T04:42:58.6329859Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6330579Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6420935Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T04:42:58.6479324Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6480146Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6584418Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-25T04:42:58.6644115Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6644891Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6760127Z Entering 'third_party/flatbuffers'
2025-04-25T04:42:58.6824479Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6825236Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6923400Z Entering 'third_party/fmt'
2025-04-25T04:42:58.6986044Z url.https://github.com/.insteadof
2025-04-25T04:42:58.6986764Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7077418Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-25T04:42:58.7139794Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7140518Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7232685Z Entering 'third_party/gloo'
2025-04-25T04:42:58.7294996Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7295720Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7387056Z Entering 'third_party/googletest'
2025-04-25T04:42:58.7449133Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7450012Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7541018Z Entering 'third_party/ideep'
2025-04-25T04:42:58.7601519Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7602277Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7690908Z Entering 'third_party/ideep/mkl-dnn'
2025-04-25T04:42:58.7750787Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7751527Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7860428Z Entering 'third_party/ittapi'
2025-04-25T04:42:58.7926082Z url.https://github.com/.insteadof
2025-04-25T04:42:58.7926813Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8023708Z Entering 'third_party/kineto'
2025-04-25T04:42:58.8086638Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8087391Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8174923Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T04:42:58.8234776Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8235437Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8320671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T04:42:58.8380639Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8381384Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8474043Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T04:42:58.8532347Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8533088Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8621333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T04:42:58.8679882Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8680638Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8768999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T04:42:58.8828147Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8828866Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8911471Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T04:42:58.8971650Z url.https://github.com/.insteadof
2025-04-25T04:42:58.8972387Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9067962Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T04:42:58.9126421Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9127157Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9215540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T04:42:58.9273501Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9274170Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9362877Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T04:42:58.9422102Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9422826Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9514908Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T04:42:58.9574057Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9574747Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9670331Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T04:42:58.9732017Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9732687Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9822564Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T04:42:58.9882018Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9882775Z url.https://github.com/.insteadof
2025-04-25T04:42:58.9979735Z Entering 'third_party/kleidiai'
2025-04-25T04:42:59.0042109Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0043144Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0137130Z Entering 'third_party/mimalloc'
2025-04-25T04:42:59.0200397Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0201117Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0290754Z Entering 'third_party/nlohmann'
2025-04-25T04:42:59.0352644Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0353315Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0444599Z Entering 'third_party/onnx'
2025-04-25T04:42:59.0507442Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0508185Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0628763Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-25T04:42:59.0689653Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0690344Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0791288Z Entering 'third_party/opentelemetry-cpp'
2025-04-25T04:42:59.0854310Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0854988Z url.https://github.com/.insteadof
2025-04-25T04:42:59.0943994Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T04:42:59.1001701Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1002400Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1087994Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T04:42:59.1145106Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1145815Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1233306Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T04:42:59.1290841Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1291575Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1378000Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T04:42:59.1434936Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1435682Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1527136Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T04:42:59.1584512Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1585180Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1670908Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T04:42:59.1730314Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1731052Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1818463Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T04:42:59.1877090Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1878682Z url.https://github.com/.insteadof
2025-04-25T04:42:59.1960575Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T04:42:59.2020567Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2021299Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2114202Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T04:42:59.2173688Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2174408Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2271689Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T04:42:59.2331294Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2332040Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2471407Z Entering 'third_party/pocketfft'
2025-04-25T04:42:59.2535517Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2536277Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2628474Z Entering 'third_party/protobuf'
2025-04-25T04:42:59.2692509Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2693306Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2787996Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-25T04:42:59.2849539Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2850270Z url.https://github.com/.insteadof
2025-04-25T04:42:59.2941058Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-25T04:42:59.3000550Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3001258Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3097705Z Entering 'third_party/psimd'
2025-04-25T04:42:59.3161080Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3161775Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3252754Z Entering 'third_party/pthreadpool'
2025-04-25T04:42:59.3314391Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3315159Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3406577Z Entering 'third_party/pybind11'
2025-04-25T04:42:59.3467909Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3468722Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3564696Z Entering 'third_party/python-peachpy'
2025-04-25T04:42:59.3627795Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3628547Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3718140Z Entering 'third_party/sleef'
2025-04-25T04:42:59.3780873Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3781635Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3872382Z Entering 'third_party/tensorpipe'
2025-04-25T04:42:59.3934783Z url.https://github.com/.insteadof
2025-04-25T04:42:59.3935505Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4024321Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-25T04:42:59.4082645Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4083446Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4170304Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-25T04:42:59.4229325Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4230058Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4314880Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-25T04:42:59.4373078Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4373737Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4462358Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T04:42:59.4520500Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4521149Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4604081Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T04:42:59.4662214Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4662939Z url.https://github.com/.insteadof
2025-04-25T04:42:59.4806205Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2025-04-25T04:42:59.5488189Z Entering 'android/libs/fbjni'
2025-04-25T04:42:59.5587112Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-04-25T04:42:59.5648370Z Entering 'third_party/FP16'
2025-04-25T04:42:59.5751094Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-04-25T04:42:59.5809374Z Entering 'third_party/FXdiv'
2025-04-25T04:42:59.5911987Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-04-25T04:42:59.5970805Z Entering 'third_party/NNPACK'
2025-04-25T04:42:59.6072045Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-04-25T04:42:59.6130133Z Entering 'third_party/NVTX'
2025-04-25T04:42:59.6232268Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-04-25T04:42:59.6290099Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-25T04:42:59.6396746Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-04-25T04:42:59.6456089Z Entering 'third_party/XNNPACK'
2025-04-25T04:42:59.6558637Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-04-25T04:42:59.6651289Z Entering 'third_party/benchmark'
2025-04-25T04:42:59.6751186Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-04-25T04:42:59.6809667Z Entering 'third_party/composable_kernel'
2025-04-25T04:42:59.6912267Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-04-25T04:42:59.6984669Z Entering 'third_party/cpp-httplib'
2025-04-25T04:42:59.7083338Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-04-25T04:42:59.7141166Z Entering 'third_party/cpuinfo'
2025-04-25T04:42:59.7245208Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-04-25T04:42:59.7303197Z Entering 'third_party/cudnn_frontend'
2025-04-25T04:42:59.7407415Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-04-25T04:42:59.7465544Z Entering 'third_party/cutlass'
2025-04-25T04:42:59.7569065Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-04-25T04:42:59.7645400Z Entering 'third_party/eigen'
2025-04-25T04:42:59.7747834Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2025-04-25T04:42:59.7811226Z Entering 'third_party/fbgemm'
2025-04-25T04:42:59.7915647Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-04-25T04:42:59.7969909Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-25T04:42:59.8066504Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2025-04-25T04:42:59.8120048Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T04:42:59.8220871Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2025-04-25T04:42:59.8276440Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-25T04:42:59.8376462Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2025-04-25T04:42:59.8445747Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-25T04:42:59.8542875Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2025-04-25T04:42:59.8596832Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T04:42:59.8697833Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2025-04-25T04:42:59.8758122Z Entering 'third_party/flash-attention'
2025-04-25T04:42:59.8861603Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-04-25T04:42:59.8918250Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T04:42:59.9018674Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-04-25T04:42:59.9088572Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-25T04:42:59.9188833Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-04-25T04:42:59.9270586Z Entering 'third_party/flatbuffers'
2025-04-25T04:42:59.9372880Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-04-25T04:42:59.9436731Z Entering 'third_party/fmt'
2025-04-25T04:42:59.9539010Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-04-25T04:42:59.9597209Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-25T04:42:59.9698532Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-04-25T04:42:59.9757741Z Entering 'third_party/gloo'
2025-04-25T04:42:59.9857787Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-04-25T04:42:59.9915663Z Entering 'third_party/googletest'
2025-04-25T04:43:00.0019087Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-04-25T04:43:00.0079731Z Entering 'third_party/ideep'
2025-04-25T04:43:00.0182925Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-04-25T04:43:00.0236889Z Entering 'third_party/ideep/mkl-dnn'
2025-04-25T04:43:00.0333881Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-04-25T04:43:00.0410031Z Entering 'third_party/ittapi'
2025-04-25T04:43:00.0510738Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-04-25T04:43:00.0572076Z Entering 'third_party/kineto'
2025-04-25T04:43:00.0675265Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-04-25T04:43:00.0731826Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T04:43:00.0831690Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-04-25T04:43:00.0881084Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T04:43:00.0979796Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-04-25T04:43:00.1037414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T04:43:00.1136752Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-04-25T04:43:00.1191017Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T04:43:00.1293833Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-04-25T04:43:00.1349044Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T04:43:00.1448278Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-04-25T04:43:00.1498268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T04:43:00.1599936Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-04-25T04:43:00.1659718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T04:43:00.1759087Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-04-25T04:43:00.1813548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T04:43:00.1912019Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-04-25T04:43:00.1965866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T04:43:00.2064717Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-04-25T04:43:00.2120074Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T04:43:00.2217778Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-04-25T04:43:00.2277473Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T04:43:00.2374442Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-04-25T04:43:00.2427158Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T04:43:00.2528270Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-04-25T04:43:00.2590612Z Entering 'third_party/kleidiai'
2025-04-25T04:43:00.2693581Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-04-25T04:43:00.2751801Z Entering 'third_party/mimalloc'
2025-04-25T04:43:00.2852992Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-04-25T04:43:00.2912056Z Entering 'third_party/nlohmann'
2025-04-25T04:43:00.3013668Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-04-25T04:43:00.3076942Z Entering 'third_party/onnx'
2025-04-25T04:43:00.3179163Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-04-25T04:43:00.3267176Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-25T04:43:00.3363809Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-04-25T04:43:00.3427868Z Entering 'third_party/opentelemetry-cpp'
2025-04-25T04:43:00.3544312Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-04-25T04:43:00.3601759Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T04:43:00.3703324Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-04-25T04:43:00.3756474Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T04:43:00.3855031Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-04-25T04:43:00.3908106Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T04:43:00.4005231Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-04-25T04:43:00.4057556Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T04:43:00.4154457Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-04-25T04:43:00.4209494Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T04:43:00.4308124Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-04-25T04:43:00.4360798Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T04:43:00.4457460Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-04-25T04:43:00.4509820Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T04:43:00.4607181Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-04-25T04:43:00.4654123Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T04:43:00.4750961Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-04-25T04:43:00.4808439Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T04:43:00.4902945Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-04-25T04:43:00.4962172Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T04:43:00.5055992Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-04-25T04:43:00.5154812Z Entering 'third_party/pocketfft'
2025-04-25T04:43:00.5255267Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-04-25T04:43:00.5312303Z Entering 'third_party/protobuf'
2025-04-25T04:43:00.5409630Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-04-25T04:43:00.5469224Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-25T04:43:00.5565465Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-04-25T04:43:00.5618601Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-25T04:43:00.5713003Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-04-25T04:43:00.5772635Z Entering 'third_party/psimd'
2025-04-25T04:43:00.5873921Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-04-25T04:43:00.5931052Z Entering 'third_party/pthreadpool'
2025-04-25T04:43:00.6031413Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-04-25T04:43:00.6087728Z Entering 'third_party/pybind11'
2025-04-25T04:43:00.6192561Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-04-25T04:43:00.6252461Z Entering 'third_party/python-peachpy'
2025-04-25T04:43:00.6350834Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-04-25T04:43:00.6406138Z Entering 'third_party/sleef'
2025-04-25T04:43:00.6504782Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-04-25T04:43:00.6566913Z Entering 'third_party/tensorpipe'
2025-04-25T04:43:00.6669872Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-04-25T04:43:00.6724506Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-25T04:43:00.6819576Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-04-25T04:43:00.6871218Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-25T04:43:00.6970197Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-04-25T04:43:00.7022666Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-25T04:43:00.7117889Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-04-25T04:43:00.7171478Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T04:43:00.7266374Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-04-25T04:43:00.7313719Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T04:43:00.7410459Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-04-25T04:43:00.7999660Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2025-04-25T04:43:00.8666530Z Entering 'android/libs/fbjni'
2025-04-25T04:43:00.8775623Z Entering 'third_party/FP16'
2025-04-25T04:43:00.8886414Z Entering 'third_party/FXdiv'
2025-04-25T04:43:00.8991765Z Entering 'third_party/NNPACK'
2025-04-25T04:43:00.9099815Z Entering 'third_party/NVTX'
2025-04-25T04:43:00.9210090Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-25T04:43:00.9318090Z Entering 'third_party/XNNPACK'
2025-04-25T04:43:00.9462931Z Entering 'third_party/benchmark'
2025-04-25T04:43:00.9571386Z Entering 'third_party/composable_kernel'
2025-04-25T04:43:00.9694686Z Entering 'third_party/cpp-httplib'
2025-04-25T04:43:00.9800393Z Entering 'third_party/cpuinfo'
2025-04-25T04:43:00.9908809Z Entering 'third_party/cudnn_frontend'
2025-04-25T04:43:01.0017079Z Entering 'third_party/cutlass'
2025-04-25T04:43:01.0140323Z Entering 'third_party/eigen'
2025-04-25T04:43:01.0250096Z Entering 'third_party/fbgemm'
2025-04-25T04:43:01.0354098Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-25T04:43:01.0453410Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T04:43:01.0554041Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-25T04:43:01.0669567Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-25T04:43:01.0768287Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T04:43:01.0872577Z Entering 'third_party/flash-attention'
2025-04-25T04:43:01.0978458Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T04:43:01.1091320Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-25T04:43:01.1216406Z Entering 'third_party/flatbuffers'
2025-04-25T04:43:01.1329433Z Entering 'third_party/fmt'
2025-04-25T04:43:01.1435275Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-25T04:43:01.1542139Z Entering 'third_party/gloo'
2025-04-25T04:43:01.1652048Z Entering 'third_party/googletest'
2025-04-25T04:43:01.1759274Z Entering 'third_party/ideep'
2025-04-25T04:43:01.1861573Z Entering 'third_party/ideep/mkl-dnn'
2025-04-25T04:43:01.1981164Z Entering 'third_party/ittapi'
2025-04-25T04:43:01.2088864Z Entering 'third_party/kineto'
2025-04-25T04:43:01.2190627Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T04:43:01.2286087Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T04:43:01.2388860Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T04:43:01.2487861Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T04:43:01.2592374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T04:43:01.2688284Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T04:43:01.2798138Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T04:43:01.2898985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T04:43:01.3002139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T04:43:01.3103689Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T04:43:01.3212555Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T04:43:01.3310832Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T04:43:01.3414989Z Entering 'third_party/kleidiai'
2025-04-25T04:43:01.3521013Z Entering 'third_party/mimalloc'
2025-04-25T04:43:01.3625834Z Entering 'third_party/nlohmann'
2025-04-25T04:43:01.3732643Z Entering 'third_party/onnx'
2025-04-25T04:43:01.3867420Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-25T04:43:01.3979118Z Entering 'third_party/opentelemetry-cpp'
2025-04-25T04:43:01.4083694Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T04:43:01.4181468Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T04:43:01.4278753Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T04:43:01.4376280Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T04:43:01.4475513Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T04:43:01.4573132Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T04:43:01.4669458Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T04:43:01.4762181Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T04:43:01.4864390Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T04:43:01.4967706Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T04:43:01.5112988Z Entering 'third_party/pocketfft'
2025-04-25T04:43:01.5218853Z Entering 'third_party/protobuf'
2025-04-25T04:43:01.5327293Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-25T04:43:01.5424162Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-25T04:43:01.5530702Z Entering 'third_party/psimd'
2025-04-25T04:43:01.5635445Z Entering 'third_party/pthreadpool'
2025-04-25T04:43:01.5741715Z Entering 'third_party/pybind11'
2025-04-25T04:43:01.5847885Z Entering 'third_party/python-peachpy'
2025-04-25T04:43:01.5951102Z Entering 'third_party/sleef'
2025-04-25T04:43:01.6057157Z Entering 'third_party/tensorpipe'
2025-04-25T04:43:01.6158349Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-25T04:43:01.6256984Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-25T04:43:01.6355817Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-25T04:43:01.6456723Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T04:43:01.6551164Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T04:43:01.6699492Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2025-04-25T04:43:01.7387012Z Entering 'android/libs/fbjni'
2025-04-25T04:43:01.7506183Z Entering 'third_party/FP16'
2025-04-25T04:43:01.7621238Z Entering 'third_party/FXdiv'
2025-04-25T04:43:01.7732239Z Entering 'third_party/NNPACK'
2025-04-25T04:43:01.7845044Z Entering 'third_party/NVTX'
2025-04-25T04:43:01.7956180Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-25T04:43:01.8070419Z Entering 'third_party/XNNPACK'
2025-04-25T04:43:01.8216972Z Entering 'third_party/benchmark'
2025-04-25T04:43:01.8326681Z Entering 'third_party/composable_kernel'
2025-04-25T04:43:01.8452855Z Entering 'third_party/cpp-httplib'
2025-04-25T04:43:01.8564457Z Entering 'third_party/cpuinfo'
2025-04-25T04:43:01.8674546Z Entering 'third_party/cudnn_frontend'
2025-04-25T04:43:01.8782768Z Entering 'third_party/cutlass'
2025-04-25T04:43:01.8914426Z Entering 'third_party/eigen'
2025-04-25T04:43:01.9028269Z Entering 'third_party/fbgemm'
2025-04-25T04:43:01.9138416Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-25T04:43:01.9243637Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T04:43:01.9348171Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-25T04:43:01.9466514Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-25T04:43:01.9571187Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T04:43:01.9687081Z Entering 'third_party/flash-attention'
2025-04-25T04:43:01.9792131Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T04:43:01.9912476Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-25T04:43:02.0050185Z Entering 'third_party/flatbuffers'
2025-04-25T04:43:02.0170963Z Entering 'third_party/fmt'
2025-04-25T04:43:02.0279957Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-25T04:43:02.0392599Z Entering 'third_party/gloo'
2025-04-25T04:43:02.0504812Z Entering 'third_party/googletest'
2025-04-25T04:43:02.0619389Z Entering 'third_party/ideep'
2025-04-25T04:43:02.0727880Z Entering 'third_party/ideep/mkl-dnn'
2025-04-25T04:43:02.0854056Z Entering 'third_party/ittapi'
2025-04-25T04:43:02.0963311Z Entering 'third_party/kineto'
2025-04-25T04:43:02.1070499Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T04:43:02.1166532Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T04:43:02.1271973Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T04:43:02.1378131Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T04:43:02.1481520Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T04:43:02.1582453Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T04:43:02.1695786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T04:43:02.1798750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T04:43:02.1902045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T04:43:02.2009739Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T04:43:02.2118890Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T04:43:02.2223377Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T04:43:02.2335713Z Entering 'third_party/kleidiai'
2025-04-25T04:43:02.2448168Z Entering 'third_party/mimalloc'
2025-04-25T04:43:02.2557765Z Entering 'third_party/nlohmann'
2025-04-25T04:43:02.2673983Z Entering 'third_party/onnx'
2025-04-25T04:43:02.2818341Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-25T04:43:02.2935416Z Entering 'third_party/opentelemetry-cpp'
2025-04-25T04:43:02.3046071Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T04:43:02.3146493Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T04:43:02.3247817Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T04:43:02.3347692Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T04:43:02.3449951Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T04:43:02.3548913Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T04:43:02.3647893Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T04:43:02.3744143Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T04:43:02.3847509Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T04:43:02.3955087Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T04:43:02.4106379Z Entering 'third_party/pocketfft'
2025-04-25T04:43:02.4214988Z Entering 'third_party/protobuf'
2025-04-25T04:43:02.4328187Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-25T04:43:02.4428879Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-25T04:43:02.4537844Z Entering 'third_party/psimd'
2025-04-25T04:43:02.4646020Z Entering 'third_party/pthreadpool'
2025-04-25T04:43:02.4755246Z Entering 'third_party/pybind11'
2025-04-25T04:43:02.4864853Z Entering 'third_party/python-peachpy'
2025-04-25T04:43:02.4972351Z Entering 'third_party/sleef'
2025-04-25T04:43:02.5081964Z Entering 'third_party/tensorpipe'
2025-04-25T04:43:02.5187392Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-25T04:43:02.5287246Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-25T04:43:02.5386253Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-25T04:43:02.5487346Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T04:43:02.5582258Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T04:43:02.5730460Z ##[endgroup]
2025-04-25T04:43:02.5909780Z [command]/usr/bin/git log -1 --format=%H
2025-04-25T04:43:02.6023316Z b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:43:02.6415821Z Prepare all required actions
2025-04-25T04:43:02.6416736Z Getting action download info
2025-04-25T04:43:02.7855161Z ##[group]Run ./.github/actions/setup-rocm
2025-04-25T04:43:02.7855648Z env:
2025-04-25T04:43:02.7855962Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:02.7856337Z ##[endgroup]
2025-04-25T04:43:02.7894966Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-04-25T04:43:02.7895821Z # ignore expansion of "docker ps -q" since it could be empty
2025-04-25T04:43:02.7896463Z # shellcheck disable=SC2046
2025-04-25T04:43:02.7896957Z docker stop $(docker ps -q) || true
2025-04-25T04:43:02.7897472Z # Prune all stopped containers.
2025-04-25T04:43:02.7897942Z docker container prune -f
2025-04-25T04:43:02.7946654Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:02.7947223Z env:
2025-04-25T04:43:02.7947543Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:02.7947991Z ##[endgroup]
2025-04-25T04:43:02.8685442Z "docker stop" requires at least 1 argument.
2025-04-25T04:43:02.8686153Z See 'docker stop --help'.
2025-04-25T04:43:02.8686486Z 
2025-04-25T04:43:02.8686784Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2025-04-25T04:43:02.8687279Z 
2025-04-25T04:43:02.8687483Z Stop one or more running containers
2025-04-25T04:43:02.9037098Z Total reclaimed space: 0B
2025-04-25T04:43:02.9133145Z ##[group]Run cat /etc/os-release || true
2025-04-25T04:43:02.9133788Z cat /etc/os-release || true
2025-04-25T04:43:02.9134384Z cat /etc/apt/sources.list.d/rocm.list || true
2025-04-25T04:43:02.9135041Z cat /opt/rocm/.info/version || true
2025-04-25T04:43:02.9135609Z whoami
2025-04-25T04:43:02.9194616Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:02.9195272Z env:
2025-04-25T04:43:02.9195673Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:02.9196131Z ##[endgroup]
2025-04-25T04:43:02.9314675Z PRETTY_NAME="Ubuntu 22.04.4 LTS"
2025-04-25T04:43:02.9315159Z NAME="Ubuntu"
2025-04-25T04:43:02.9315560Z VERSION_ID="22.04"
2025-04-25T04:43:02.9315955Z VERSION="22.04.4 LTS (Jammy Jellyfish)"
2025-04-25T04:43:02.9316432Z VERSION_CODENAME=jammy
2025-04-25T04:43:02.9316800Z ID=ubuntu
2025-04-25T04:43:02.9317096Z ID_LIKE=debian
2025-04-25T04:43:02.9317514Z HOME_URL="https://www.ubuntu.com/"
2025-04-25T04:43:02.9318053Z SUPPORT_URL="https://help.ubuntu.com/"
2025-04-25T04:43:02.9318637Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2025-04-25T04:43:02.9319617Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2025-04-25T04:43:02.9320387Z UBUNTU_CODENAME=jammy
2025-04-25T04:43:02.9343783Z deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3.3 jammy main
2025-04-25T04:43:02.9371491Z 6.3.3-74
2025-04-25T04:43:02.9417267Z pytorchci
2025-04-25T04:43:02.9470436Z ##[group]Run dpkg -l | grep -E "  amdgpu"
2025-04-25T04:43:02.9471133Z dpkg -l | grep -E "  amdgpu"
2025-04-25T04:43:02.9527268Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:02.9527926Z env:
2025-04-25T04:43:02.9528298Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:02.9528750Z ##[endgroup]
2025-04-25T04:43:02.9958919Z ii  amdgpu-core                            1:6.3.60303-2119913.22.04               all          Core meta package for unified amdgpu driver.
2025-04-25T04:43:02.9960317Z ii  amdgpu-dkms                            1:6.10.5.60303-2119913.22.04            all          amdgpu driver in DKMS format.
2025-04-25T04:43:02.9961626Z ii  amdgpu-dkms-firmware                   1:6.10.5.60303-2119913.22.04            all          firmware blobs used by amdgpu driver in DKMS format
2025-04-25T04:43:02.9962919Z ii  amdgpu-install                         6.3.60303-2119913.22.04                 all          AMDGPU driver repository and installer
2025-04-25T04:43:03.0020942Z ##[group]Run rocm-smi
2025-04-25T04:43:03.0021447Z rocm-smi
2025-04-25T04:43:03.0080678Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:03.0081483Z env:
2025-04-25T04:43:03.0081949Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:03.0082485Z ##[endgroup]
2025-04-25T04:43:03.2008936Z 
2025-04-25T04:43:03.2009083Z 
2025-04-25T04:43:03.2009684Z ========================================= ROCm System Management Interface =========================================
2025-04-25T04:43:03.2010692Z =================================================== Concise Info ===================================================
2025-04-25T04:43:03.2011751Z Device  Node  IDs              Temp    Power  Partitions          SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
2025-04-25T04:43:03.2013335Z               (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                                   
2025-04-25T04:43:03.2014175Z ====================================================================================================================
2025-04-25T04:43:03.2015499Z 0       4     0x740c,   57586  42.0°C  94.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-04-25T04:43:03.2016686Z 1       5     0x740c,   45873  43.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-04-25T04:43:03.2017801Z 2       2     0x740c,   51627  40.0°C  95.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-04-25T04:43:03.2018881Z 3       3     0x740c,   64489  37.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-04-25T04:43:03.2019978Z 4       8     0x740c,   30939  38.0°C  98.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-04-25T04:43:03.2021049Z 5       9     0x740c,   8466   36.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-04-25T04:43:03.2022141Z 6       6     0x740c,   41154  40.0°C  93.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-04-25T04:43:03.2023870Z 7       7     0x740c,   63755  36.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-04-25T04:43:03.2024648Z ====================================================================================================================
2025-04-25T04:43:03.2025358Z =============================================== End of ROCm SMI Log ================================================
2025-04-25T04:43:03.2172831Z ##[group]Run rocminfo
2025-04-25T04:43:03.2173228Z rocminfo
2025-04-25T04:43:03.2218051Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:03.2218627Z env:
2025-04-25T04:43:03.2218958Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:03.2219333Z ##[endgroup]
2025-04-25T04:43:03.3658845Z ROCk module version 6.10.5 is loaded
2025-04-25T04:43:03.3659576Z =====================    
2025-04-25T04:43:03.3660079Z HSA System Attributes    
2025-04-25T04:43:03.3660531Z =====================    
2025-04-25T04:43:03.3660950Z Runtime Version:         1.14
2025-04-25T04:43:03.3661552Z Runtime Ext Version:     1.6
2025-04-25T04:43:03.3662060Z System Timestamp Freq.:  1000.000000MHz
2025-04-25T04:43:03.3662889Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-04-25T04:43:03.3663769Z Machine Model:           LARGE                              
2025-04-25T04:43:03.3664614Z System Endianness:       LITTLE                             
2025-04-25T04:43:03.3665361Z Mwaitx:                  DISABLED
2025-04-25T04:43:03.3665857Z DMAbuf Support:          YES
2025-04-25T04:43:03.3666142Z 
2025-04-25T04:43:03.3666289Z ==========               
2025-04-25T04:43:03.3666701Z HSA Agents               
2025-04-25T04:43:03.3667104Z ==========               
2025-04-25T04:43:03.3667476Z *******                  
2025-04-25T04:43:03.3667863Z Agent 1                  
2025-04-25T04:43:03.3668237Z *******                  
2025-04-25T04:43:03.3668735Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-04-25T04:43:03.3669921Z   Uuid:                    CPU-XX                             
2025-04-25T04:43:03.3670587Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-04-25T04:43:03.3671263Z   Vendor Name:             CPU                                
2025-04-25T04:43:03.3671904Z   Feature:                 None specified                     
2025-04-25T04:43:03.3672581Z   Profile:                 FULL_PROFILE                       
2025-04-25T04:43:03.3673254Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.3673930Z   Max Queue Number:        0(0x0)                             
2025-04-25T04:43:03.3674580Z   Queue Min Size:          0(0x0)                             
2025-04-25T04:43:03.3675203Z   Queue Max Size:          0(0x0)                             
2025-04-25T04:43:03.3675834Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.3676421Z   Node:                    0                                  
2025-04-25T04:43:03.3677055Z   Device Type:             CPU                                
2025-04-25T04:43:03.3677618Z   Cache Info:              
2025-04-25T04:43:03.3678092Z     L1:                      32768(0x8000) KB                   
2025-04-25T04:43:03.3678680Z   Chip ID:                 0(0x0)                             
2025-04-25T04:43:03.3679296Z   ASIC Revision:           0(0x0)                             
2025-04-25T04:43:03.3680141Z   Cacheline Size:          64(0x40)                           
2025-04-25T04:43:03.3680791Z   Max Clock Freq. (MHz):   2000                               
2025-04-25T04:43:03.3681405Z   BDFID:                   0                                  
2025-04-25T04:43:03.3682017Z   Internal Node ID:        0                                  
2025-04-25T04:43:03.3682788Z   Compute Unit:            64                                 
2025-04-25T04:43:03.3683521Z   SIMDs per CU:            0                                  
2025-04-25T04:43:03.3684282Z   Shader Engines:          0                                  
2025-04-25T04:43:03.3685396Z   Shader Arrs. per Eng.:   0                                  
2025-04-25T04:43:03.3686101Z   WatchPts on Addr. Ranges:1                                  
2025-04-25T04:43:03.3686733Z   Memory Properties:       
2025-04-25T04:43:03.3687179Z   Features:                None
2025-04-25T04:43:03.3687634Z   Pool Info:               
2025-04-25T04:43:03.3688052Z     Pool 1                   
2025-04-25T04:43:03.3688608Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.3689269Z       Size:                    528196732(0x1f7ba47c) KB           
2025-04-25T04:43:03.3689898Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3690561Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3691245Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:43:03.3691949Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3692643Z       Accessible by all:       TRUE                               
2025-04-25T04:43:03.3693223Z     Pool 2                   
2025-04-25T04:43:03.3693750Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.3694482Z       Size:                    528196732(0x1f7ba47c) KB           
2025-04-25T04:43:03.3695268Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3695994Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3696701Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:43:03.3697397Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3698078Z       Accessible by all:       TRUE                               
2025-04-25T04:43:03.3698647Z     Pool 3                   
2025-04-25T04:43:03.3699184Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-04-25T04:43:03.3700339Z       Size:                    528196732(0x1f7ba47c) KB           
2025-04-25T04:43:03.3701066Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3701743Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3702442Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:43:03.3703143Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3703847Z       Accessible by all:       TRUE                               
2025-04-25T04:43:03.3704432Z     Pool 4                   
2025-04-25T04:43:03.3704970Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.3705594Z       Size:                    528196732(0x1f7ba47c) KB           
2025-04-25T04:43:03.3706225Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3706906Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3707606Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:43:03.3708605Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3709284Z       Accessible by all:       TRUE                               
2025-04-25T04:43:03.3709884Z   ISA Info:                
2025-04-25T04:43:03.3710446Z *******                  
2025-04-25T04:43:03.3710863Z Agent 2                  
2025-04-25T04:43:03.3711256Z *******                  
2025-04-25T04:43:03.3711722Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-04-25T04:43:03.3712356Z   Uuid:                    CPU-XX                             
2025-04-25T04:43:03.3713002Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-04-25T04:43:03.3713686Z   Vendor Name:             CPU                                
2025-04-25T04:43:03.3714370Z   Feature:                 None specified                     
2025-04-25T04:43:03.3715164Z   Profile:                 FULL_PROFILE                       
2025-04-25T04:43:03.3716141Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.3716816Z   Max Queue Number:        0(0x0)                             
2025-04-25T04:43:03.3717480Z   Queue Min Size:          0(0x0)                             
2025-04-25T04:43:03.3718112Z   Queue Max Size:          0(0x0)                             
2025-04-25T04:43:03.3718751Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.3719339Z   Node:                    1                                  
2025-04-25T04:43:03.3720061Z   Device Type:             CPU                                
2025-04-25T04:43:03.3720626Z   Cache Info:              
2025-04-25T04:43:03.3721096Z     L1:                      32768(0x8000) KB                   
2025-04-25T04:43:03.3721679Z   Chip ID:                 0(0x0)                             
2025-04-25T04:43:03.3722288Z   ASIC Revision:           0(0x0)                             
2025-04-25T04:43:03.3722956Z   Cacheline Size:          64(0x40)                           
2025-04-25T04:43:03.3723633Z   Max Clock Freq. (MHz):   2000                               
2025-04-25T04:43:03.3724257Z   BDFID:                   0                                  
2025-04-25T04:43:03.3724859Z   Internal Node ID:        1                                  
2025-04-25T04:43:03.3725509Z   Compute Unit:            64                                 
2025-04-25T04:43:03.3726146Z   SIMDs per CU:            0                                  
2025-04-25T04:43:03.3726772Z   Shader Engines:          0                                  
2025-04-25T04:43:03.3727437Z   Shader Arrs. per Eng.:   0                                  
2025-04-25T04:43:03.3728111Z   WatchPts on Addr. Ranges:1                                  
2025-04-25T04:43:03.3728729Z   Memory Properties:       
2025-04-25T04:43:03.3729158Z   Features:                None
2025-04-25T04:43:03.3729607Z   Pool Info:               
2025-04-25T04:43:03.3730020Z     Pool 1                   
2025-04-25T04:43:03.3730916Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.3731576Z       Size:                    528455492(0x1f7f9744) KB           
2025-04-25T04:43:03.3732203Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3732890Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3733590Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:43:03.3734296Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3734965Z       Accessible by all:       TRUE                               
2025-04-25T04:43:03.3735561Z     Pool 2                   
2025-04-25T04:43:03.3736102Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.3736733Z       Size:                    528455492(0x1f7f9744) KB           
2025-04-25T04:43:03.3737364Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3738050Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3738748Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:43:03.3739484Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3740199Z       Accessible by all:       TRUE                               
2025-04-25T04:43:03.3740801Z     Pool 3                   
2025-04-25T04:43:03.3741336Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-04-25T04:43:03.3741991Z       Size:                    528455492(0x1f7f9744) KB           
2025-04-25T04:43:03.3742617Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3743297Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3743997Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:43:03.3744717Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3745755Z       Accessible by all:       TRUE                               
2025-04-25T04:43:03.3746343Z     Pool 4                   
2025-04-25T04:43:03.3746888Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.3747514Z       Size:                    528455492(0x1f7f9744) KB           
2025-04-25T04:43:03.3748160Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3748821Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3749526Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:43:03.3750244Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3750918Z       Accessible by all:       TRUE                               
2025-04-25T04:43:03.3751519Z   ISA Info:                
2025-04-25T04:43:03.3751934Z *******                  
2025-04-25T04:43:03.3752339Z Agent 3                  
2025-04-25T04:43:03.3752719Z *******                  
2025-04-25T04:43:03.3753203Z   Name:                    gfx90a                             
2025-04-25T04:43:03.3753809Z   Uuid:                    GPU-270fbd25d3852d1b               
2025-04-25T04:43:03.3754483Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:43:03.3755161Z   Vendor Name:             AMD                                
2025-04-25T04:43:03.3755802Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:43:03.3756448Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:43:03.3757100Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.3757769Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:43:03.3758415Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:43:03.3759062Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:43:03.3760013Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.3760959Z   Node:                    2                                  
2025-04-25T04:43:03.3761580Z   Device Type:             GPU                                
2025-04-25T04:43:03.3762150Z   Cache Info:              
2025-04-25T04:43:03.3762619Z     L1:                      16(0x10) KB                        
2025-04-25T04:43:03.3763164Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:43:03.3763740Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:43:03.3764366Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:43:03.3765035Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:43:03.3765749Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:43:03.3766394Z   BDFID:                   12800                              
2025-04-25T04:43:03.3767040Z   Internal Node ID:        2                                  
2025-04-25T04:43:03.3767689Z   Compute Unit:            104                                
2025-04-25T04:43:03.3768352Z   SIMDs per CU:            4                                  
2025-04-25T04:43:03.3768993Z   Shader Engines:          8                                  
2025-04-25T04:43:03.3769681Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:43:03.3770367Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:43:03.3771053Z   Coherent Host Access:    FALSE                              
2025-04-25T04:43:03.3771673Z   Memory Properties:       
2025-04-25T04:43:03.3772159Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:43:03.3772775Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:43:03.3773444Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:43:03.3774123Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.3774743Z   Workgroup Max Size per Dimension:
2025-04-25T04:43:03.3775268Z     x                        1024(0x400)                        
2025-04-25T04:43:03.3776238Z     y                        1024(0x400)                        
2025-04-25T04:43:03.3776776Z     z                        1024(0x400)                        
2025-04-25T04:43:03.3777383Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:43:03.3778053Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:43:03.3778725Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.3779307Z   Grid Max Size per Dimension:
2025-04-25T04:43:03.3779797Z     x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3780348Z     y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3780891Z     z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3781533Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:43:03.3791658Z   Packet Processor uCode:: 92                                 
2025-04-25T04:43:03.3792518Z   SDMA engine uCode::      9                                  
2025-04-25T04:43:03.3793246Z   IOMMU Support::          None                               
2025-04-25T04:43:03.3793836Z   Pool Info:               
2025-04-25T04:43:03.3794273Z     Pool 1                   
2025-04-25T04:43:03.3794830Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.3795494Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.3796137Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3796792Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3797490Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.3798190Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3798865Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3799458Z     Pool 2                   
2025-04-25T04:43:03.3800593Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.3801286Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.3801925Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3802595Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3803298Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.3804003Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3804677Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3805260Z     Pool 3                   
2025-04-25T04:43:03.3805794Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.3806432Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.3807067Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3807758Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3808454Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.3809162Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3809842Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3810427Z     Pool 4                   
2025-04-25T04:43:03.3810927Z       Segment:                 GROUP                              
2025-04-25T04:43:03.3811529Z       Size:                    64(0x40) KB                        
2025-04-25T04:43:03.3812144Z       Allocatable:             FALSE                              
2025-04-25T04:43:03.3812803Z       Alloc Granule:           0KB                                
2025-04-25T04:43:03.3813502Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:43:03.3814202Z       Alloc Alignment:         0KB                                
2025-04-25T04:43:03.3815246Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3815825Z   ISA Info:                
2025-04-25T04:43:03.3816259Z     ISA 1                    
2025-04-25T04:43:03.3817037Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:43:03.3817823Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:43:03.3818617Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:43:03.3819439Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.3820191Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.3820839Z       Fast f16:                TRUE                               
2025-04-25T04:43:03.3821527Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.3822182Z       Workgroup Max Size per Dimension:
2025-04-25T04:43:03.3822776Z         x                        1024(0x400)                        
2025-04-25T04:43:03.3823438Z         y                        1024(0x400)                        
2025-04-25T04:43:03.3824069Z         z                        1024(0x400)                        
2025-04-25T04:43:03.3824666Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.3825302Z       Grid Max Size per Dimension:
2025-04-25T04:43:03.3825803Z         x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3826349Z         y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3826887Z         z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3827557Z       FBarrier Max Size:       32                                 
2025-04-25T04:43:03.3828143Z *******                  
2025-04-25T04:43:03.3828537Z Agent 4                  
2025-04-25T04:43:03.3828958Z *******                  
2025-04-25T04:43:03.3829421Z   Name:                    gfx90a                             
2025-04-25T04:43:03.3830337Z   Uuid:                    GPU-da59425c5973e0bc               
2025-04-25T04:43:03.3831009Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:43:03.3831689Z   Vendor Name:             AMD                                
2025-04-25T04:43:03.3832324Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:43:03.3832969Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:43:03.3833624Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.3834277Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:43:03.3834935Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:43:03.3835569Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:43:03.3836202Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.3836817Z   Node:                    3                                  
2025-04-25T04:43:03.3837488Z   Device Type:             GPU                                
2025-04-25T04:43:03.3838086Z   Cache Info:              
2025-04-25T04:43:03.3838567Z     L1:                      16(0x10) KB                        
2025-04-25T04:43:03.3839130Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:43:03.3839815Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:43:03.3840450Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:43:03.3841106Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:43:03.3841773Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:43:03.3842396Z   BDFID:                   13568                              
2025-04-25T04:43:03.3843015Z   Internal Node ID:        3                                  
2025-04-25T04:43:03.3843677Z   Compute Unit:            104                                
2025-04-25T04:43:03.3844656Z   SIMDs per CU:            4                                  
2025-04-25T04:43:03.3845308Z   Shader Engines:          8                                  
2025-04-25T04:43:03.3845967Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:43:03.3846661Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:43:03.3847375Z   Coherent Host Access:    FALSE                              
2025-04-25T04:43:03.3847990Z   Memory Properties:       
2025-04-25T04:43:03.3848479Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:43:03.3849088Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:43:03.3849775Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:43:03.3850445Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.3851075Z   Workgroup Max Size per Dimension:
2025-04-25T04:43:03.3851585Z     x                        1024(0x400)                        
2025-04-25T04:43:03.3852157Z     y                        1024(0x400)                        
2025-04-25T04:43:03.3852690Z     z                        1024(0x400)                        
2025-04-25T04:43:03.3853286Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:43:03.3853965Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:43:03.3854619Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.3855214Z   Grid Max Size per Dimension:
2025-04-25T04:43:03.3855692Z     x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3856261Z     y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3856804Z     z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3857425Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:43:03.3858165Z   Packet Processor uCode:: 92                                 
2025-04-25T04:43:03.3858859Z   SDMA engine uCode::      9                                  
2025-04-25T04:43:03.3859878Z   IOMMU Support::          None                               
2025-04-25T04:43:03.3860469Z   Pool Info:               
2025-04-25T04:43:03.3860897Z     Pool 1                   
2025-04-25T04:43:03.3861587Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.3862298Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.3862941Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3863607Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3864320Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.3865040Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3865721Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3866321Z     Pool 2                   
2025-04-25T04:43:03.3866851Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.3867521Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.3868133Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3868790Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3869492Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.3870185Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3870868Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3871443Z     Pool 3                   
2025-04-25T04:43:03.3871975Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.3872594Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.3873228Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3873893Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3874876Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.3875581Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3876257Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3876840Z     Pool 4                   
2025-04-25T04:43:03.3877336Z       Segment:                 GROUP                              
2025-04-25T04:43:03.3877938Z       Size:                    64(0x40) KB                        
2025-04-25T04:43:03.3878563Z       Allocatable:             FALSE                              
2025-04-25T04:43:03.3879215Z       Alloc Granule:           0KB                                
2025-04-25T04:43:03.3880289Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:43:03.3880981Z       Alloc Alignment:         0KB                                
2025-04-25T04:43:03.3881661Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3882266Z   ISA Info:                
2025-04-25T04:43:03.3882683Z     ISA 1                    
2025-04-25T04:43:03.3883238Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:43:03.3883954Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:43:03.3884642Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:43:03.3885329Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.3885955Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.3886524Z       Fast f16:                TRUE                               
2025-04-25T04:43:03.3887103Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.3887662Z       Workgroup Max Size per Dimension:
2025-04-25T04:43:03.3888135Z         x                        1024(0x400)                        
2025-04-25T04:43:03.3888978Z         y                        1024(0x400)                        
2025-04-25T04:43:03.3889460Z         z                        1024(0x400)                        
2025-04-25T04:43:03.3889991Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.3890503Z       Grid Max Size per Dimension:
2025-04-25T04:43:03.3890937Z         x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3891414Z         y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3891890Z         z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3892440Z       FBarrier Max Size:       32                                 
2025-04-25T04:43:03.3892928Z *******                  
2025-04-25T04:43:03.3893290Z Agent 5                  
2025-04-25T04:43:03.3893628Z *******                  
2025-04-25T04:43:03.3894033Z   Name:                    gfx90a                             
2025-04-25T04:43:03.3894567Z   Uuid:                    GPU-3faa9ae703e0aa3c               
2025-04-25T04:43:03.3895171Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:43:03.3895766Z   Vendor Name:             AMD                                
2025-04-25T04:43:03.3896331Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:43:03.3896903Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:43:03.3897481Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.3898073Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:43:03.3898644Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:43:03.3899209Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:43:03.3899768Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.3900299Z   Node:                    4                                  
2025-04-25T04:43:03.3900834Z   Device Type:             GPU                                
2025-04-25T04:43:03.3901655Z   Cache Info:              
2025-04-25T04:43:03.3902081Z     L1:                      16(0x10) KB                        
2025-04-25T04:43:03.3902566Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:43:03.3903080Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:43:03.3903619Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:43:03.3904207Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:43:03.3904791Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:43:03.3905328Z   BDFID:                   4352                               
2025-04-25T04:43:03.3905877Z   Internal Node ID:        4                                  
2025-04-25T04:43:03.3906451Z   Compute Unit:            104                                
2025-04-25T04:43:03.3932087Z   SIMDs per CU:            4                                  
2025-04-25T04:43:03.3932774Z   Shader Engines:          8                                  
2025-04-25T04:43:03.3933402Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:43:03.3934008Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:43:03.3934634Z   Coherent Host Access:    FALSE                              
2025-04-25T04:43:03.3935182Z   Memory Properties:       
2025-04-25T04:43:03.3935610Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:43:03.3936165Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:43:03.3936757Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:43:03.3937346Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.3937880Z   Workgroup Max Size per Dimension:
2025-04-25T04:43:03.3938364Z     x                        1024(0x400)                        
2025-04-25T04:43:03.3938862Z     y                        1024(0x400)                        
2025-04-25T04:43:03.3939735Z     z                        1024(0x400)                        
2025-04-25T04:43:03.3940302Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:43:03.3940890Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:43:03.3941470Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.3942007Z   Grid Max Size per Dimension:
2025-04-25T04:43:03.3942453Z     x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3942939Z     y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3943415Z     z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3943968Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:43:03.3944628Z   Packet Processor uCode:: 92                                 
2025-04-25T04:43:03.3945255Z   SDMA engine uCode::      9                                  
2025-04-25T04:43:03.3945837Z   IOMMU Support::          None                               
2025-04-25T04:43:03.3946375Z   Pool Info:               
2025-04-25T04:43:03.3946760Z     Pool 1                   
2025-04-25T04:43:03.3947255Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.3947845Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.3948402Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3948994Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3949607Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.3950246Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3950850Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3951372Z     Pool 2                   
2025-04-25T04:43:03.3951848Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.3952749Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.3953341Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3953929Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3954545Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.3955169Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3955782Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3956301Z     Pool 3                   
2025-04-25T04:43:03.3956762Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.3957315Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.3957861Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.3958450Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.3959077Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.3960058Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.3960702Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3961236Z     Pool 4                   
2025-04-25T04:43:03.3961706Z       Segment:                 GROUP                              
2025-04-25T04:43:03.3962236Z       Size:                    64(0x40) KB                        
2025-04-25T04:43:03.3962786Z       Allocatable:             FALSE                              
2025-04-25T04:43:03.3963372Z       Alloc Granule:           0KB                                
2025-04-25T04:43:03.3964015Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:43:03.3964651Z       Alloc Alignment:         0KB                                
2025-04-25T04:43:03.3965258Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.3965785Z   ISA Info:                
2025-04-25T04:43:03.3966479Z     ISA 1                    
2025-04-25T04:43:03.3966981Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:43:03.3967621Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:43:03.3968233Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:43:03.3968838Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.3969459Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.3970038Z       Fast f16:                TRUE                               
2025-04-25T04:43:03.3970623Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.3971183Z       Workgroup Max Size per Dimension:
2025-04-25T04:43:03.3971656Z         x                        1024(0x400)                        
2025-04-25T04:43:03.3972140Z         y                        1024(0x400)                        
2025-04-25T04:43:03.3972628Z         z                        1024(0x400)                        
2025-04-25T04:43:03.3973155Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.3973744Z       Grid Max Size per Dimension:
2025-04-25T04:43:03.3974178Z         x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3974663Z         y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3975138Z         z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.3975679Z       FBarrier Max Size:       32                                 
2025-04-25T04:43:03.3976180Z *******                  
2025-04-25T04:43:03.3976539Z Agent 6                  
2025-04-25T04:43:03.3976882Z *******                  
2025-04-25T04:43:03.3977286Z   Name:                    gfx90a                             
2025-04-25T04:43:03.3977829Z   Uuid:                    GPU-c930cceae5e422f1               
2025-04-25T04:43:03.3978723Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:43:03.3979330Z   Vendor Name:             AMD                                
2025-04-25T04:43:03.3979884Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:43:03.3980460Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:43:03.3981032Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.3981621Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:43:03.3982195Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:43:03.3982756Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:43:03.3983317Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.3983836Z   Node:                    5                                  
2025-04-25T04:43:03.3984364Z   Device Type:             GPU                                
2025-04-25T04:43:03.3984866Z   Cache Info:              
2025-04-25T04:43:03.3985305Z     L1:                      16(0x10) KB                        
2025-04-25T04:43:03.3985799Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:43:03.3986295Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:43:03.3986845Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:43:03.3987417Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:43:03.3988003Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:43:03.3988551Z   BDFID:                   5120                               
2025-04-25T04:43:03.3989128Z   Internal Node ID:        5                                  
2025-04-25T04:43:03.3989710Z   Compute Unit:            104                                
2025-04-25T04:43:03.3990270Z   SIMDs per CU:            4                                  
2025-04-25T04:43:03.3990843Z   Shader Engines:          8                                  
2025-04-25T04:43:03.3991686Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:43:03.3992311Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:43:03.3992920Z   Coherent Host Access:    FALSE                              
2025-04-25T04:43:03.3993458Z   Memory Properties:       
2025-04-25T04:43:03.3993891Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:43:03.3994429Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:43:03.3995024Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:43:03.3995607Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.3996152Z   Workgroup Max Size per Dimension:
2025-04-25T04:43:03.3996613Z     x                        1024(0x400)                        
2025-04-25T04:43:03.3997098Z     y                        1024(0x400)                        
2025-04-25T04:43:03.3997558Z     z                        1024(0x400)                        
2025-04-25T04:43:03.3998105Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:43:03.3998692Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:43:03.3999268Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.3999899Z   Grid Max Size per Dimension:
2025-04-25T04:43:03.4000312Z     x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4000808Z     y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4001278Z     z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4001829Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:43:03.4002463Z   Packet Processor uCode:: 92                                 
2025-04-25T04:43:03.4003077Z   SDMA engine uCode::      9                                  
2025-04-25T04:43:03.4003664Z   IOMMU Support::          None                               
2025-04-25T04:43:03.4004520Z   Pool Info:               
2025-04-25T04:43:03.4004913Z     Pool 1                   
2025-04-25T04:43:03.4005391Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.4005969Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4006535Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4007115Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4007737Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4008355Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4008963Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4009473Z     Pool 2                   
2025-04-25T04:43:03.4009945Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.4010502Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4011061Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4011647Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4012254Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4012871Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4013468Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4013979Z     Pool 3                   
2025-04-25T04:43:03.4014434Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.4014975Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4015515Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4016089Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4016698Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4017605Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4018213Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4018720Z     Pool 4                   
2025-04-25T04:43:03.4019162Z       Segment:                 GROUP                              
2025-04-25T04:43:03.4019688Z       Size:                    64(0x40) KB                        
2025-04-25T04:43:03.4020223Z       Allocatable:             FALSE                              
2025-04-25T04:43:03.4020814Z       Alloc Granule:           0KB                                
2025-04-25T04:43:03.4021418Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:43:03.4022040Z       Alloc Alignment:         0KB                                
2025-04-25T04:43:03.4022631Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4023146Z   ISA Info:                
2025-04-25T04:43:03.4023513Z     ISA 1                    
2025-04-25T04:43:03.4024016Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:43:03.4024650Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:43:03.4025251Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:43:03.4025856Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4026469Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4027040Z       Fast f16:                TRUE                               
2025-04-25T04:43:03.4027614Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.4028153Z       Workgroup Max Size per Dimension:
2025-04-25T04:43:03.4028624Z         x                        1024(0x400)                        
2025-04-25T04:43:03.4029102Z         y                        1024(0x400)                        
2025-04-25T04:43:03.4029565Z         z                        1024(0x400)                        
2025-04-25T04:43:03.4030347Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.4030867Z       Grid Max Size per Dimension:
2025-04-25T04:43:03.4031307Z         x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4031779Z         y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4032251Z         z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4032808Z       FBarrier Max Size:       32                                 
2025-04-25T04:43:03.4033323Z *******                  
2025-04-25T04:43:03.4033677Z Agent 7                  
2025-04-25T04:43:03.4034023Z *******                  
2025-04-25T04:43:03.4034422Z   Name:                    gfx90a                             
2025-04-25T04:43:03.4034970Z   Uuid:                    GPU-4a1afa394ae4f5e3               
2025-04-25T04:43:03.4035558Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:43:03.4036167Z   Vendor Name:             AMD                                
2025-04-25T04:43:03.4036736Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:43:03.4037298Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:43:03.4037879Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.4038463Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:43:03.4039034Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:43:03.4039692Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:43:03.4040257Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.4040789Z   Node:                    6                                  
2025-04-25T04:43:03.4041317Z   Device Type:             GPU                                
2025-04-25T04:43:03.4041819Z   Cache Info:              
2025-04-25T04:43:03.4042223Z     L1:                      16(0x10) KB                        
2025-04-25T04:43:03.4043021Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:43:03.4043556Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:43:03.4044102Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:43:03.4044686Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:43:03.4045285Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:43:03.4045850Z   BDFID:                   44544                              
2025-04-25T04:43:03.4046396Z   Internal Node ID:        6                                  
2025-04-25T04:43:03.4046995Z   Compute Unit:            104                                
2025-04-25T04:43:03.4047574Z   SIMDs per CU:            4                                  
2025-04-25T04:43:03.4048151Z   Shader Engines:          8                                  
2025-04-25T04:43:03.4048754Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:43:03.4049393Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:43:03.4050022Z   Coherent Host Access:    FALSE                              
2025-04-25T04:43:03.4050572Z   Memory Properties:       
2025-04-25T04:43:03.4051008Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:43:03.4051550Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:43:03.4052161Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:43:03.4052779Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.4053324Z   Workgroup Max Size per Dimension:
2025-04-25T04:43:03.4053800Z     x                        1024(0x400)                        
2025-04-25T04:43:03.4054285Z     y                        1024(0x400)                        
2025-04-25T04:43:03.4054764Z     z                        1024(0x400)                        
2025-04-25T04:43:03.4055300Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:43:03.4056212Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:43:03.4056815Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.4057336Z   Grid Max Size per Dimension:
2025-04-25T04:43:03.4057760Z     x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4058237Z     y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4058716Z     z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4059269Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:43:03.4059910Z   Packet Processor uCode:: 92                                 
2025-04-25T04:43:03.4060530Z   SDMA engine uCode::      9                                  
2025-04-25T04:43:03.4061121Z   IOMMU Support::          None                               
2025-04-25T04:43:03.4061644Z   Pool Info:               
2025-04-25T04:43:03.4062020Z     Pool 1                   
2025-04-25T04:43:03.4062530Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.4063100Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4063671Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4064266Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4064893Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4065531Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4066137Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4066658Z     Pool 2                   
2025-04-25T04:43:03.4067128Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.4067714Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4068283Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4069143Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4069779Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4070407Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4071020Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4071530Z     Pool 3                   
2025-04-25T04:43:03.4072000Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.4072550Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4073107Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4073701Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4074315Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4074946Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4075573Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4076097Z     Pool 4                   
2025-04-25T04:43:03.4076542Z       Segment:                 GROUP                              
2025-04-25T04:43:03.4077087Z       Size:                    64(0x40) KB                        
2025-04-25T04:43:03.4077647Z       Allocatable:             FALSE                              
2025-04-25T04:43:03.4078232Z       Alloc Granule:           0KB                                
2025-04-25T04:43:03.4078851Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:43:03.4079691Z       Alloc Alignment:         0KB                                
2025-04-25T04:43:03.4080310Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4080823Z   ISA Info:                
2025-04-25T04:43:03.4081194Z     ISA 1                    
2025-04-25T04:43:03.4081681Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:43:03.4082636Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:43:03.4083252Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:43:03.4083858Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4084497Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4085069Z       Fast f16:                TRUE                               
2025-04-25T04:43:03.4085653Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.4086206Z       Workgroup Max Size per Dimension:
2025-04-25T04:43:03.4086682Z         x                        1024(0x400)                        
2025-04-25T04:43:03.4087173Z         y                        1024(0x400)                        
2025-04-25T04:43:03.4087637Z         z                        1024(0x400)                        
2025-04-25T04:43:03.4088169Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.4088718Z       Grid Max Size per Dimension:
2025-04-25T04:43:03.4089166Z         x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4089650Z         y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4090139Z         z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4090699Z       FBarrier Max Size:       32                                 
2025-04-25T04:43:03.4091209Z *******                  
2025-04-25T04:43:03.4091574Z Agent 8                  
2025-04-25T04:43:03.4091918Z *******                  
2025-04-25T04:43:03.4092326Z   Name:                    gfx90a                             
2025-04-25T04:43:03.4092858Z   Uuid:                    GPU-4c986e30208c5b32               
2025-04-25T04:43:03.4093455Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:43:03.4094063Z   Vendor Name:             AMD                                
2025-04-25T04:43:03.4094919Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:43:03.4095512Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:43:03.4096098Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.4096694Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:43:03.4097265Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:43:03.4097837Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:43:03.4098404Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.4098942Z   Node:                    7                                  
2025-04-25T04:43:03.4099487Z   Device Type:             GPU                                
2025-04-25T04:43:03.4099986Z   Cache Info:              
2025-04-25T04:43:03.4100402Z     L1:                      16(0x10) KB                        
2025-04-25T04:43:03.4100901Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:43:03.4101424Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:43:03.4101972Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:43:03.4102558Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:43:03.4103148Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:43:03.4103684Z   BDFID:                   45824                              
2025-04-25T04:43:03.4104245Z   Internal Node ID:        7                                  
2025-04-25T04:43:03.4104818Z   Compute Unit:            104                                
2025-04-25T04:43:03.4105389Z   SIMDs per CU:            4                                  
2025-04-25T04:43:03.4105949Z   Shader Engines:          8                                  
2025-04-25T04:43:03.4106542Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:43:03.4107153Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:43:03.4108070Z   Coherent Host Access:    FALSE                              
2025-04-25T04:43:03.4108617Z   Memory Properties:       
2025-04-25T04:43:03.4109033Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:43:03.4109581Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:43:03.4110176Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:43:03.4110776Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.4111326Z   Workgroup Max Size per Dimension:
2025-04-25T04:43:03.4111773Z     x                        1024(0x400)                        
2025-04-25T04:43:03.4112255Z     y                        1024(0x400)                        
2025-04-25T04:43:03.4112720Z     z                        1024(0x400)                        
2025-04-25T04:43:03.4113256Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:43:03.4113848Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:43:03.4114468Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.4114989Z   Grid Max Size per Dimension:
2025-04-25T04:43:03.4115412Z     x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4115900Z     y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4116384Z     z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4116949Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:43:03.4117585Z   Packet Processor uCode:: 92                                 
2025-04-25T04:43:03.4118208Z   SDMA engine uCode::      9                                  
2025-04-25T04:43:03.4118803Z   IOMMU Support::          None                               
2025-04-25T04:43:03.4119322Z   Pool Info:               
2025-04-25T04:43:03.4119804Z     Pool 1                   
2025-04-25T04:43:03.4120284Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.4121179Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4121756Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4122361Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4122985Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4123620Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4124292Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4124868Z     Pool 2                   
2025-04-25T04:43:03.4125402Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.4126077Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4126632Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4127235Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4127882Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4128501Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4129108Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4129629Z     Pool 3                   
2025-04-25T04:43:03.4130096Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.4130656Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4131215Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4131815Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4132425Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4133055Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4133659Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4134500Z     Pool 4                   
2025-04-25T04:43:03.4135010Z       Segment:                 GROUP                              
2025-04-25T04:43:03.4135556Z       Size:                    64(0x40) KB                        
2025-04-25T04:43:03.4136111Z       Allocatable:             FALSE                              
2025-04-25T04:43:03.4136707Z       Alloc Granule:           0KB                                
2025-04-25T04:43:03.4137338Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:43:03.4137968Z       Alloc Alignment:         0KB                                
2025-04-25T04:43:03.4138661Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4139189Z   ISA Info:                
2025-04-25T04:43:03.4139564Z     ISA 1                    
2025-04-25T04:43:03.4140060Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:43:03.4140717Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:43:03.4141436Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:43:03.4142056Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4142694Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4143272Z       Fast f16:                TRUE                               
2025-04-25T04:43:03.4143859Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.4144431Z       Workgroup Max Size per Dimension:
2025-04-25T04:43:03.4144900Z         x                        1024(0x400)                        
2025-04-25T04:43:03.4145391Z         y                        1024(0x400)                        
2025-04-25T04:43:03.4145900Z         z                        1024(0x400)                        
2025-04-25T04:43:03.4146447Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.4146984Z       Grid Max Size per Dimension:
2025-04-25T04:43:03.4147691Z         x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4148197Z         y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4148681Z         z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4149229Z       FBarrier Max Size:       32                                 
2025-04-25T04:43:03.4149737Z *******                  
2025-04-25T04:43:03.4150102Z Agent 9                  
2025-04-25T04:43:03.4150439Z *******                  
2025-04-25T04:43:03.4150885Z   Name:                    gfx90a                             
2025-04-25T04:43:03.4151439Z   Uuid:                    GPU-39d022f9dbb71856               
2025-04-25T04:43:03.4152055Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:43:03.4152665Z   Vendor Name:             AMD                                
2025-04-25T04:43:03.4153237Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:43:03.4153858Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:43:03.4154446Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.4155056Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:43:03.4155632Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:43:03.4156215Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:43:03.4156793Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.4157320Z   Node:                    8                                  
2025-04-25T04:43:03.4157865Z   Device Type:             GPU                                
2025-04-25T04:43:03.4158407Z   Cache Info:              
2025-04-25T04:43:03.4158838Z     L1:                      16(0x10) KB                        
2025-04-25T04:43:03.4159332Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:43:03.4160255Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:43:03.4160816Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:43:03.4161399Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:43:03.4161995Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:43:03.4162536Z   BDFID:                   36352                              
2025-04-25T04:43:03.4163098Z   Internal Node ID:        8                                  
2025-04-25T04:43:03.4163675Z   Compute Unit:            104                                
2025-04-25T04:43:03.4164248Z   SIMDs per CU:            4                                  
2025-04-25T04:43:03.4164828Z   Shader Engines:          8                                  
2025-04-25T04:43:03.4165417Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:43:03.4166060Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:43:03.4166696Z   Coherent Host Access:    FALSE                              
2025-04-25T04:43:03.4167248Z   Memory Properties:       
2025-04-25T04:43:03.4167668Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:43:03.4168223Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:43:03.4168815Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:43:03.4169430Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.4169988Z   Workgroup Max Size per Dimension:
2025-04-25T04:43:03.4170444Z     x                        1024(0x400)                        
2025-04-25T04:43:03.4170926Z     y                        1024(0x400)                        
2025-04-25T04:43:03.4171393Z     z                        1024(0x400)                        
2025-04-25T04:43:03.4171932Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:43:03.4172523Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:43:03.4173449Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.4173991Z   Grid Max Size per Dimension:
2025-04-25T04:43:03.4174410Z     x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4174913Z     y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4175401Z     z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4175974Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:43:03.4176622Z   Packet Processor uCode:: 92                                 
2025-04-25T04:43:03.4177258Z   SDMA engine uCode::      9                                  
2025-04-25T04:43:03.4177883Z   IOMMU Support::          None                               
2025-04-25T04:43:03.4178407Z   Pool Info:               
2025-04-25T04:43:03.4178811Z     Pool 1                   
2025-04-25T04:43:03.4179292Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.4179904Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4180496Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4181122Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4181779Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4182424Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4183057Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4183589Z     Pool 2                   
2025-04-25T04:43:03.4184082Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.4184659Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4185234Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4185836Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4186484Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4187398Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4188011Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4188541Z     Pool 3                   
2025-04-25T04:43:03.4189010Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.4189583Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4190142Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4190746Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4191380Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4192011Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4192638Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4193155Z     Pool 4                   
2025-04-25T04:43:03.4193625Z       Segment:                 GROUP                              
2025-04-25T04:43:03.4194161Z       Size:                    64(0x40) KB                        
2025-04-25T04:43:03.4194724Z       Allocatable:             FALSE                              
2025-04-25T04:43:03.4195328Z       Alloc Granule:           0KB                                
2025-04-25T04:43:03.4195945Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:43:03.4196578Z       Alloc Alignment:         0KB                                
2025-04-25T04:43:03.4197192Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4197727Z   ISA Info:                
2025-04-25T04:43:03.4198092Z     ISA 1                    
2025-04-25T04:43:03.4198584Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:43:03.4199240Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:43:03.4200459Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:43:03.4201103Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4201736Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4202328Z       Fast f16:                TRUE                               
2025-04-25T04:43:03.4202915Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.4203489Z       Workgroup Max Size per Dimension:
2025-04-25T04:43:03.4203971Z         x                        1024(0x400)                        
2025-04-25T04:43:03.4204471Z         y                        1024(0x400)                        
2025-04-25T04:43:03.4204988Z         z                        1024(0x400)                        
2025-04-25T04:43:03.4205618Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.4206265Z       Grid Max Size per Dimension:
2025-04-25T04:43:03.4206804Z         x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4207411Z         y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4207984Z         z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4208695Z       FBarrier Max Size:       32                                 
2025-04-25T04:43:03.4209314Z *******                  
2025-04-25T04:43:03.4209743Z Agent 10                 
2025-04-25T04:43:03.4210177Z *******                  
2025-04-25T04:43:03.4210654Z   Name:                    gfx90a                             
2025-04-25T04:43:03.4211317Z   Uuid:                    GPU-24bb52fc4f169fad               
2025-04-25T04:43:03.4211687Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:43:03.4212015Z   Vendor Name:             AMD                                
2025-04-25T04:43:03.4212293Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:43:03.4212560Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:43:03.4212993Z   Float Round Mode:        NEAR                               
2025-04-25T04:43:03.4213265Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:43:03.4213539Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:43:03.4213805Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:43:03.4214074Z   Queue Type:              MULTI                              
2025-04-25T04:43:03.4214317Z   Node:                    9                                  
2025-04-25T04:43:03.4214577Z   Device Type:             GPU                                
2025-04-25T04:43:03.4214822Z   Cache Info:              
2025-04-25T04:43:03.4215015Z     L1:                      16(0x10) KB                        
2025-04-25T04:43:03.4215254Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:43:03.4215489Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:43:03.4215773Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:43:03.4216045Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:43:03.4216329Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:43:03.4216594Z   BDFID:                   37632                              
2025-04-25T04:43:03.4216850Z   Internal Node ID:        9                                  
2025-04-25T04:43:03.4217129Z   Compute Unit:            104                                
2025-04-25T04:43:03.4217388Z   SIMDs per CU:            4                                  
2025-04-25T04:43:03.4217669Z   Shader Engines:          8                                  
2025-04-25T04:43:03.4217946Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:43:03.4218240Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:43:03.4218542Z   Coherent Host Access:    FALSE                              
2025-04-25T04:43:03.4218801Z   Memory Properties:       
2025-04-25T04:43:03.4219130Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:43:03.4219392Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:43:03.4219685Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:43:03.4219963Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.4220227Z   Workgroup Max Size per Dimension:
2025-04-25T04:43:03.4220443Z     x                        1024(0x400)                        
2025-04-25T04:43:03.4220681Z     y                        1024(0x400)                        
2025-04-25T04:43:03.4220908Z     z                        1024(0x400)                        
2025-04-25T04:43:03.4221151Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:43:03.4221438Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:43:03.4221719Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.4221998Z   Grid Max Size per Dimension:
2025-04-25T04:43:03.4222202Z     x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4222447Z     y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4222691Z     z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4222954Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:43:03.4223275Z   Packet Processor uCode:: 92                                 
2025-04-25T04:43:03.4223579Z   SDMA engine uCode::      9                                  
2025-04-25T04:43:03.4223884Z   IOMMU Support::          None                               
2025-04-25T04:43:03.4224142Z   Pool Info:               
2025-04-25T04:43:03.4224340Z     Pool 1                   
2025-04-25T04:43:03.4224582Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:43:03.4224864Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4225288Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4225575Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4225890Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4226196Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4226502Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4226762Z     Pool 2                   
2025-04-25T04:43:03.4226993Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:43:03.4227275Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4227540Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4227833Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4228124Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4228442Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4228738Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4228986Z     Pool 3                   
2025-04-25T04:43:03.4229209Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:43:03.4229473Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:43:03.4229746Z       Allocatable:             TRUE                               
2025-04-25T04:43:03.4230032Z       Alloc Granule:           4KB                                
2025-04-25T04:43:03.4230332Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:43:03.4230636Z       Alloc Alignment:         4KB                                
2025-04-25T04:43:03.4230926Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4231185Z     Pool 4                   
2025-04-25T04:43:03.4231397Z       Segment:                 GROUP                              
2025-04-25T04:43:03.4231782Z       Size:                    64(0x40) KB                        
2025-04-25T04:43:03.4232049Z       Allocatable:             FALSE                              
2025-04-25T04:43:03.4232336Z       Alloc Granule:           0KB                                
2025-04-25T04:43:03.4232632Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:43:03.4232938Z       Alloc Alignment:         0KB                                
2025-04-25T04:43:03.4233237Z       Accessible by all:       FALSE                              
2025-04-25T04:43:03.4233489Z   ISA Info:                
2025-04-25T04:43:03.4233675Z     ISA 1                    
2025-04-25T04:43:03.4233906Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:43:03.4234220Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:43:03.4234516Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:43:03.4234834Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4235147Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:43:03.4235434Z       Fast f16:                TRUE                               
2025-04-25T04:43:03.4235722Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:43:03.4235995Z       Workgroup Max Size per Dimension:
2025-04-25T04:43:03.4236229Z         x                        1024(0x400)                        
2025-04-25T04:43:03.4236538Z         y                        1024(0x400)                        
2025-04-25T04:43:03.4236795Z         z                        1024(0x400)                        
2025-04-25T04:43:03.4237072Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:43:03.4237337Z       Grid Max Size per Dimension:
2025-04-25T04:43:03.4237570Z         x                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4237938Z         y                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4238189Z         z                        4294967295(0xffffffff)             
2025-04-25T04:43:03.4238453Z       FBarrier Max Size:       32                                 
2025-04-25T04:43:03.4238724Z *** Done ***             
2025-04-25T04:43:03.4255950Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-04-25T04:43:03.4256287Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-04-25T04:43:03.4256835Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-04-25T04:43:03.4257338Z if [[ $ngpu -eq 0 ]]; then
2025-04-25T04:43:03.4257602Z     echo "Error: Failed to detect any GPUs on the runner"
2025-04-25T04:43:03.4257882Z     echo "$msg"
2025-04-25T04:43:03.4258059Z     exit 1
2025-04-25T04:43:03.4258232Z fi
2025-04-25T04:43:03.4258406Z if [[ $ngpu -eq 1 ]]; then
2025-04-25T04:43:03.4258748Z     echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs"
2025-04-25T04:43:03.4259077Z     echo "$msg"
2025-04-25T04:43:03.4259263Z     exit 1
2025-04-25T04:43:03.4259434Z fi
2025-04-25T04:43:03.4281724Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:03.4282007Z env:
2025-04-25T04:43:03.4282165Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:03.4282365Z ##[endgroup]
2025-04-25T04:43:03.6061791Z ##[group]Run pytorch/pytorch/.github/actions/diskspace-cleanup@main
2025-04-25T04:43:03.6062520Z with:
2025-04-25T04:43:03.6062888Z   diskspace-cutoff: 70
2025-04-25T04:43:03.6063303Z env:
2025-04-25T04:43:03.6063652Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:03.6064088Z ##[endgroup]
2025-04-25T04:43:03.6130711Z ##[group]Run set -ex
2025-04-25T04:43:03.6131237Z set -ex
2025-04-25T04:43:03.6131640Z diskspace_cutoff=70
2025-04-25T04:43:03.6132782Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-04-25T04:43:03.6133520Z if [ ! -d "$docker_root_dir" ]; then
2025-04-25T04:43:03.6134457Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-04-25T04:43:03.6135303Z     exit 0
2025-04-25T04:43:03.6135696Z fi
2025-04-25T04:43:03.6136423Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-04-25T04:43:03.6137949Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-04-25T04:43:03.6139251Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-04-25T04:43:03.6139903Z     docker system prune -af
2025-04-25T04:43:03.6140811Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-04-25T04:43:03.6141841Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-04-25T04:43:03.6142853Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2025-04-25T04:43:03.6143784Z         echo "$msg"
2025-04-25T04:43:03.6144236Z         exit 1
2025-04-25T04:43:03.6144660Z     else
2025-04-25T04:43:03.6170362Z         difference=$((diskspace - diskspace_new))
2025-04-25T04:43:03.6171151Z         echo "Diskspace saved: $difference percent"
2025-04-25T04:43:03.6171723Z     fi
2025-04-25T04:43:03.6172056Z fi
2025-04-25T04:43:03.6220498Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:03.6221102Z env:
2025-04-25T04:43:03.6221434Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:03.6221832Z ##[endgroup]
2025-04-25T04:43:03.6308195Z + diskspace_cutoff=70
2025-04-25T04:43:03.6316988Z ++ docker info -f '{{.DockerRootDir}}'
2025-04-25T04:43:03.7299348Z + docker_root_dir=/media/4TB/docker-rootless
2025-04-25T04:43:03.7300053Z + '[' '!' -d /media/4TB/docker-rootless ']'
2025-04-25T04:43:03.7314322Z ++ df -H --output=pcent /media/4TB/docker-rootless
2025-04-25T04:43:03.7317065Z ++ sed -n 2p
2025-04-25T04:43:03.7321616Z ++ sed s/%//
2025-04-25T04:43:03.7323835Z ++ sed 's/ //'
2025-04-25T04:43:03.7367233Z + diskspace=20
2025-04-25T04:43:03.7368751Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-04-25T04:43:03.7370005Z + [[ 20 -ge 70 ]]
2025-04-25T04:43:03.7420881Z ##[group]Run RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-04-25T04:43:03.7421712Z RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-04-25T04:43:03.7422386Z rm -rf "${RUNNER_ARTIFACT_DIR}"
2025-04-25T04:43:03.7422965Z mkdir -p "${RUNNER_ARTIFACT_DIR}"
2025-04-25T04:43:03.7423736Z echo "RUNNER_ARTIFACT_DIR=${RUNNER_ARTIFACT_DIR}" >> "${GITHUB_ENV}"
2025-04-25T04:43:03.7424516Z 
2025-04-25T04:43:03.7425059Z RUNNER_TEST_RESULTS_DIR="${RUNNER_TEMP}/test-results"
2025-04-25T04:43:03.7425785Z rm -rf "${RUNNER_TEST_RESULTS_DIR}"
2025-04-25T04:43:03.7426379Z mkdir -p "${RUNNER_TEST_RESULTS_DIR}"
2025-04-25T04:43:03.7427198Z echo "RUNNER_TEST_RESULTS_DIR=${RUNNER_TEST_RESULTS_DIR}" >> "${GITHUB_ENV}"
2025-04-25T04:43:03.7427953Z 
2025-04-25T04:43:03.7428373Z RUNNER_DOCS_DIR="${RUNNER_TEMP}/docs"
2025-04-25T04:43:03.7428953Z rm -rf "${RUNNER_DOCS_DIR}"
2025-04-25T04:43:03.7429474Z mkdir -p "${RUNNER_DOCS_DIR}"
2025-04-25T04:43:03.7430142Z echo "RUNNER_DOCS_DIR=${RUNNER_DOCS_DIR}" >> "${GITHUB_ENV}"
2025-04-25T04:43:03.7480616Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:03.7481415Z env:
2025-04-25T04:43:03.7481829Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:03.7482259Z ##[endgroup]
2025-04-25T04:43:03.7811041Z ##[group]Run env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-04-25T04:43:03.7812070Z env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-04-25T04:43:03.7812923Z env | grep '^CI' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-04-25T04:43:03.7867726Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:03.7868377Z env:
2025-04-25T04:43:03.7868740Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:03.7869484Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:03.7870582Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:03.7871630Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:03.7872335Z ##[endgroup]
2025-04-25T04:43:03.8100980Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-04-25T04:43:03.8102326Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-04-25T04:43:03.8103248Z # Add render group for container creation.
2025-04-25T04:43:03.8103982Z render_gid=`cat /etc/group | grep render | cut -d: -f3`
2025-04-25T04:43:03.8104871Z # Ensure GPU isolation if pod is part of kubernetes setup with DEVICE_FLAG.
2025-04-25T04:43:03.8105757Z if [ -f "/etc/podinfo/gha-render-devices" ]; then
2025-04-25T04:43:03.8106505Z   DEVICE_FLAG=$(cat /etc/podinfo/gha-render-devices)
2025-04-25T04:43:03.8107128Z else
2025-04-25T04:43:03.8107543Z   DEVICE_FLAG="--device /dev/dri"
2025-04-25T04:43:03.8108036Z fi
2025-04-25T04:43:03.8108831Z # The --group-add daemon and --group-add bin are needed in the Ubuntu 24.04 and Almalinux OSs respectively.
2025-04-25T04:43:03.8110102Z # This is due to the device files (/dev/kfd & /dev/dri) being owned by video group on bare metal.
2025-04-25T04:43:03.8111697Z # This video group ID maps to subgid 1 inside the docker image due to the /etc/subgid entries.
2025-04-25T04:43:03.8112932Z # The group name corresponding to group ID 1 can change depending on the OS, so both are necessary.
2025-04-25T04:43:03.8115012Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd $DEVICE_FLAG --group-add video --group-add $render_gid --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host" >> "${GITHUB_ENV}"
2025-04-25T04:43:03.8167744Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:03.8168412Z env:
2025-04-25T04:43:03.8168774Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:03.8169509Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:03.8170592Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:03.8171696Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:03.8172458Z ##[endgroup]
2025-04-25T04:43:03.8439835Z ##[group]Run aws-actions/configure-aws-credentials@v4
2025-04-25T04:43:03.8440544Z with:
2025-04-25T04:43:03.8441211Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only
2025-04-25T04:43:03.8442162Z   aws-region: us-east-1
2025-04-25T04:43:03.8442714Z   role-duration-seconds: 18000
2025-04-25T04:43:03.8443301Z   audience: sts.amazonaws.com
2025-04-25T04:43:03.8443763Z env:
2025-04-25T04:43:03.8444100Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:03.8444831Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:03.8445893Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:03.8446898Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:03.8448683Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:03.8450226Z ##[endgroup]
2025-04-25T04:43:04.1772852Z Assuming role with OIDC
2025-04-25T04:43:04.3558496Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions
2025-04-25T04:43:04.4326746Z ##[group]Run aws-actions/amazon-ecr-login@v2
2025-04-25T04:43:04.4327444Z with:
2025-04-25T04:43:04.4327898Z   mask-password: true
2025-04-25T04:43:04.4328393Z   registry-type: private
2025-04-25T04:43:04.4328908Z   skip-logout: false
2025-04-25T04:43:04.4329346Z env:
2025-04-25T04:43:04.4329760Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:04.4330563Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:04.4331738Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:04.4332803Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:04.4334914Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:04.4336584Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:04.4337181Z   AWS_REGION: us-east-1
2025-04-25T04:43:04.4338360Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:04.4339089Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:04.4350258Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:04.4350745Z ##[endgroup]
2025-04-25T04:43:04.7605814Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:05.1284740Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2025-04-25T04:43:05.1285506Z with:
2025-04-25T04:43:05.1286555Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:05.1287686Z   docker-build-dir: .ci/docker
2025-04-25T04:43:05.1288172Z   docker-build-script: ./build.sh
2025-04-25T04:43:05.1289091Z   working-directory: .
2025-04-25T04:43:05.1289650Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:05.1290298Z   force-push: false
2025-04-25T04:43:05.1290677Z env:
2025-04-25T04:43:05.1291028Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:05.1291694Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:05.1292675Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:05.1293660Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:05.1295211Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:05.1296581Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:05.1297019Z   AWS_REGION: us-east-1
2025-04-25T04:43:05.1297564Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:05.1298154Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:05.1307492Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:05.1307883Z ##[endgroup]
2025-04-25T04:43:05.1340885Z ##[group]Run set -ex
2025-04-25T04:43:05.1341368Z set -ex
2025-04-25T04:43:05.1341741Z 
2025-04-25T04:43:05.1342369Z # If the docker build directory or the build script doesn't exist, the action will
2025-04-25T04:43:05.1343395Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2025-04-25T04:43:05.1344281Z # job could then download the pre-built image as usual
2025-04-25T04:43:05.1345135Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]]; then
2025-04-25T04:43:05.1345928Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:05.1346629Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:05.1347260Z 
2025-04-25T04:43:05.1347804Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2025-04-25T04:43:05.1348485Z   exit 0
2025-04-25T04:43:05.1348840Z else
2025-04-25T04:43:05.1349253Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:05.1349743Z fi
2025-04-25T04:43:05.1350073Z 
2025-04-25T04:43:05.1350598Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2025-04-25T04:43:05.1351499Z   # The docker image name already includes the ECR prefix and tag, so we can just
2025-04-25T04:43:05.1352331Z   # use it as it is, but first let's extract the tag
2025-04-25T04:43:05.1353078Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2025-04-25T04:43:05.1353853Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:05.1354597Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:05.1355206Z else
2025-04-25T04:43:05.1355903Z   DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2025-04-25T04:43:05.1357318Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:05.1358529Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:05.1359677Z fi
2025-04-25T04:43:05.1408807Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:05.1409831Z env:
2025-04-25T04:43:05.1410718Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:05.1411636Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:05.1413011Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:05.1414141Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:05.1415870Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:05.1417897Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:05.1446118Z   AWS_REGION: us-east-1
2025-04-25T04:43:05.1446714Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:05.1447359Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:05.1456744Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:05.1457192Z   REPO_NAME: pytorch
2025-04-25T04:43:05.1458258Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:05.1459403Z   DOCKER_BUILD_DIR: .ci/docker
2025-04-25T04:43:05.1459883Z   DOCKER_BUILD_SCRIPT: ./build.sh
2025-04-25T04:43:05.1460491Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:05.1461114Z   CUSTOM_TAG_PREFIX: 
2025-04-25T04:43:05.1461516Z ##[endgroup]
2025-04-25T04:43:05.1551513Z + [[ ! -d .ci/docker ]]
2025-04-25T04:43:05.1552153Z + [[ ! -f .ci/docker/./build.sh ]]
2025-04-25T04:43:05.1552700Z + echo skip=false
2025-04-25T04:43:05.1555050Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2025-04-25T04:43:05.1572638Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:05.1574464Z ++ awk -F '[:,]' '{print $2}'
2025-04-25T04:43:05.1632914Z + DOCKER_TAG=f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:05.1633816Z + echo docker-tag=f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:05.1635359Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:05.1698071Z ##[group]Run set +e
2025-04-25T04:43:05.1698630Z set +e
2025-04-25T04:43:05.1699155Z set -x
2025-04-25T04:43:05.1699621Z 
2025-04-25T04:43:05.1700075Z login() {
2025-04-25T04:43:05.1701089Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-04-25T04:43:05.1702072Z }
2025-04-25T04:43:05.1702445Z 
2025-04-25T04:43:05.1702819Z retry () {
2025-04-25T04:43:05.1703297Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-04-25T04:43:05.1703856Z }
2025-04-25T04:43:05.1704214Z 
2025-04-25T04:43:05.1704649Z retry login "${DOCKER_REGISTRY}"
2025-04-25T04:43:05.1705204Z 
2025-04-25T04:43:05.1705598Z START_TIME=$(date +%s)
2025-04-25T04:43:05.1706135Z # Wait up to 120 minutes
2025-04-25T04:43:05.1706794Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do
2025-04-25T04:43:05.1707672Z   # Check if image already exists, if it does then skip building it
2025-04-25T04:43:05.1708547Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2025-04-25T04:43:05.1709194Z     exit 0
2025-04-25T04:43:05.1709616Z   fi
2025-04-25T04:43:05.1710001Z 
2025-04-25T04:43:05.1710669Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2025-04-25T04:43:05.1711813Z   # use this to differentiate between the Docker build and regular build jobs. For the
2025-04-25T04:43:05.1713372Z   # latter, it will wait for the Docker images to become available before continuing
2025-04-25T04:43:05.1714311Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2025-04-25T04:43:05.1715031Z     # It's a Docker build job, let's build the image
2025-04-25T04:43:05.1715639Z     break
2025-04-25T04:43:05.1716054Z   else
2025-04-25T04:43:05.1716660Z     # It's a regular build job, wait for the image to become available
2025-04-25T04:43:05.1717387Z     sleep 300
2025-04-25T04:43:05.1718185Z   fi
2025-04-25T04:43:05.1718576Z done
2025-04-25T04:43:05.1718939Z 
2025-04-25T04:43:05.1719912Z # NB: This part requires a full checkout. Otherwise, the merge base will
2025-04-25T04:43:05.1720937Z # be empty.  The default action would be to continue rebuild the image
2025-04-25T04:43:05.1721991Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2025-04-25T04:43:05.1722929Z   # if we're on the base branch then use the parent commit
2025-04-25T04:43:05.1723674Z   MERGE_BASE=$(git rev-parse HEAD~)
2025-04-25T04:43:05.1724230Z else
2025-04-25T04:43:05.1724803Z   # otherwise we're on a PR, so use the most recent base commit
2025-04-25T04:43:05.1725622Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2025-04-25T04:43:05.1726250Z fi
2025-04-25T04:43:05.1726631Z 
2025-04-25T04:43:05.1727050Z if [[ -z "${MERGE_BASE}" ]]; then
2025-04-25T04:43:05.1727684Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:05.1728264Z 
2025-04-25T04:43:05.1729220Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2025-04-25T04:43:05.1730325Z   exit 0
2025-04-25T04:43:05.1730802Z fi
2025-04-25T04:43:05.1731238Z 
2025-04-25T04:43:05.1731880Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2025-04-25T04:43:05.1733274Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2025-04-25T04:43:05.1734274Z   exit 1
2025-04-25T04:43:05.1734657Z fi
2025-04-25T04:43:05.1735020Z 
2025-04-25T04:43:05.1735643Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2025-04-25T04:43:05.1736775Z # If no image exists but the hash is the same as the previous hash then we should error out here
2025-04-25T04:43:05.1737813Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2025-04-25T04:43:05.1739110Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2025-04-25T04:43:05.1740725Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2025-04-25T04:43:05.1741647Z fi
2025-04-25T04:43:05.1742025Z 
2025-04-25T04:43:05.1742508Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:05.1796033Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:05.1796726Z env:
2025-04-25T04:43:05.1797128Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:05.1797914Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:05.1799029Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:05.1800217Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:05.1802204Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:05.1803921Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:05.1804422Z   AWS_REGION: us-east-1
2025-04-25T04:43:05.1805059Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:05.1805751Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:05.1817443Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:05.1817952Z   DOCKER_BUILD_DIR: .ci/docker
2025-04-25T04:43:05.1818572Z   BASE_REVISION: b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:43:05.1820195Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:05.1821725Z   DOCKER_TAG: f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:05.1822497Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:05.1823509Z   DOCKER_PUSH: 
2025-04-25T04:43:05.1823915Z ##[endgroup]
2025-04-25T04:43:05.1913521Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:05.1914245Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:05.1921998Z + aws ecr get-login-password --region us-east-1
2025-04-25T04:43:05.1925606Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:06.2874716Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-04-25T04:43:06.2875859Z Configure a credential helper to remove this warning. See
2025-04-25T04:43:06.2876874Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-04-25T04:43:06.2877564Z 
2025-04-25T04:43:06.2884602Z Login Succeeded
2025-04-25T04:43:06.2941314Z ++ date +%s
2025-04-25T04:43:06.2970368Z + START_TIME=1745556186
2025-04-25T04:43:06.2979195Z ++ date +%s
2025-04-25T04:43:06.3004485Z + [[ 1745548986 -lt 1745556186 ]]
2025-04-25T04:43:06.3005943Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:07.0210922Z {
2025-04-25T04:43:07.0211545Z 	"schemaVersion": 2,
2025-04-25T04:43:07.0212569Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2025-04-25T04:43:07.0213609Z 	"config": {
2025-04-25T04:43:07.0214308Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2025-04-25T04:43:07.0215225Z 		"size": 30698,
2025-04-25T04:43:07.0216040Z 		"digest": "sha256:34cd6ce721a4f3484d87152141847442aeb9accf3a8fd11075a1d216bdb18006"
2025-04-25T04:43:07.0216938Z 	},
2025-04-25T04:43:07.0217319Z 	"layers": [
2025-04-25T04:43:07.0217706Z 		{
2025-04-25T04:43:07.0218314Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0219109Z 			"size": 28583629,
2025-04-25T04:43:07.0219930Z 			"digest": "sha256:e573979f5081f39d8f12d1ebe77f347a662dbc42b84c785396daaeca6f9d0b6d"
2025-04-25T04:43:07.0220823Z 		},
2025-04-25T04:43:07.0221182Z 		{
2025-04-25T04:43:07.0221794Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0222578Z 			"size": 1894,
2025-04-25T04:43:07.0223341Z 			"digest": "sha256:2db5ce1e0631a1e7fcc2e5c9c8c6e769189b140f77821368de7e2c4277f28e29"
2025-04-25T04:43:07.0224208Z 		},
2025-04-25T04:43:07.0224560Z 		{
2025-04-25T04:43:07.0225172Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0225964Z 			"size": 312630526,
2025-04-25T04:43:07.0226775Z 			"digest": "sha256:f1ecf534a9fdfb59bae323654b169b005a2c380e1cce50d93817d0346ac34ad4"
2025-04-25T04:43:07.0227642Z 		},
2025-04-25T04:43:07.0228005Z 		{
2025-04-25T04:43:07.0228605Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0229379Z 			"size": 703,
2025-04-25T04:43:07.0230118Z 			"digest": "sha256:5c3409aa49471c0337810f0b576dfb38e35ed8f247a07825ed05b21944efc675"
2025-04-25T04:43:07.0230962Z 		},
2025-04-25T04:43:07.0231300Z 		{
2025-04-25T04:43:07.0231922Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0232794Z 			"size": 1260,
2025-04-25T04:43:07.0233656Z 			"digest": "sha256:0f535a0b73c41d382d72aff1c8030575c276973743963a6a5a018ae684e4f688"
2025-04-25T04:43:07.0234470Z 		},
2025-04-25T04:43:07.0234821Z 		{
2025-04-25T04:43:07.0235408Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0236838Z 			"size": 484,
2025-04-25T04:43:07.0237645Z 			"digest": "sha256:c4f481217b1cf2a8fb7e57fd6e8639f5ca0ad6f65bd543bbccd015f4cc5ebfe8"
2025-04-25T04:43:07.0238511Z 		},
2025-04-25T04:43:07.0238860Z 		{
2025-04-25T04:43:07.0239765Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0240556Z 			"size": 93672332,
2025-04-25T04:43:07.0241394Z 			"digest": "sha256:4335f8ad059c7a313a53f7ba2e73a10d43c3105dc8a1d64fd3ac6238d704f6ef"
2025-04-25T04:43:07.0242896Z 		},
2025-04-25T04:43:07.0243330Z 		{
2025-04-25T04:43:07.0243960Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0244714Z 			"size": 3766,
2025-04-25T04:43:07.0245478Z 			"digest": "sha256:545f3611ba9a0c1f472938ce0af527904ea6c18d61f1a302dd1d56c59bbb363e"
2025-04-25T04:43:07.0246311Z 		},
2025-04-25T04:43:07.0246662Z 		{
2025-04-25T04:43:07.0247245Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0248098Z 			"size": 1865,
2025-04-25T04:43:07.0248926Z 			"digest": "sha256:697dcf911c268376be0cf2071df1170c51163237e40a2e9e89398961881496f0"
2025-04-25T04:43:07.0249762Z 		},
2025-04-25T04:43:07.0250129Z 		{
2025-04-25T04:43:07.0250725Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0251474Z 			"size": 701,
2025-04-25T04:43:07.0252242Z 			"digest": "sha256:7cdffbeea3da2f9187485c11a0b245cec525a103720990daf8ae5d9c9be089a4"
2025-04-25T04:43:07.0253095Z 		},
2025-04-25T04:43:07.0253456Z 		{
2025-04-25T04:43:07.0254033Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0254787Z 			"size": 2795587768,
2025-04-25T04:43:07.0255549Z 			"digest": "sha256:bf964271ff191a54c589b5115d3b24c51e94a514fdeaf99fe1fc650181099858"
2025-04-25T04:43:07.0256388Z 		},
2025-04-25T04:43:07.0256743Z 		{
2025-04-25T04:43:07.0257336Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0258070Z 			"size": 380,
2025-04-25T04:43:07.0258805Z 			"digest": "sha256:2bd40b41b145276980ca7fb47093c137f7b1fa446ac8e51a98c665d024103298"
2025-04-25T04:43:07.0259639Z 		},
2025-04-25T04:43:07.0259993Z 		{
2025-04-25T04:43:07.0260578Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0261327Z 			"size": 68638751,
2025-04-25T04:43:07.0262102Z 			"digest": "sha256:b1a7204915b7145a4851d71ff6a7971b30510018bc6aeb848508c1fe7dad9fac"
2025-04-25T04:43:07.0263080Z 		},
2025-04-25T04:43:07.0263486Z 		{
2025-04-25T04:43:07.0264093Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0264820Z 			"size": 791,
2025-04-25T04:43:07.0265549Z 			"digest": "sha256:22e97809bfdef3a4812372b779a4c1b703702f5daacf0bf1220b6a9eab60de58"
2025-04-25T04:43:07.0266423Z 		},
2025-04-25T04:43:07.0266833Z 		{
2025-04-25T04:43:07.0267519Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0268382Z 			"size": 104,
2025-04-25T04:43:07.0269131Z 			"digest": "sha256:7f692a070e087c9498ee65572ae9dfd8758312bd0e7bd550f086d07269598946"
2025-04-25T04:43:07.0269955Z 		},
2025-04-25T04:43:07.0270306Z 		{
2025-04-25T04:43:07.0270886Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0271606Z 			"size": 504,
2025-04-25T04:43:07.0272342Z 			"digest": "sha256:7d5dcba359ec17d73b3f99e4b659fe07b827a0c6e3b3935bca910c6c13294b59"
2025-04-25T04:43:07.0273186Z 		},
2025-04-25T04:43:07.0273529Z 		{
2025-04-25T04:43:07.0274103Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0274876Z 			"size": 108411751,
2025-04-25T04:43:07.0275648Z 			"digest": "sha256:26bc3b655c2c79eb5c377f4fbb043fe92beb08b16b18db034bb8f92b1f56006a"
2025-04-25T04:43:07.0276492Z 		},
2025-04-25T04:43:07.0276836Z 		{
2025-04-25T04:43:07.0277399Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0278129Z 			"size": 109,
2025-04-25T04:43:07.0278913Z 			"digest": "sha256:d3d49a239b3b7dea3fcedf79eccae347e8bfdcec79ce422cf1a7051b5f15cce3"
2025-04-25T04:43:07.0280106Z 		},
2025-04-25T04:43:07.0280840Z 		{
2025-04-25T04:43:07.0281457Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0282210Z 			"size": 1472,
2025-04-25T04:43:07.0282957Z 			"digest": "sha256:1eb0033afa635aa978fadd518569d6319c1c30ab2dd66ff677e5b57d2191b097"
2025-04-25T04:43:07.0283796Z 		},
2025-04-25T04:43:07.0284144Z 		{
2025-04-25T04:43:07.0284739Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0285838Z 			"size": 429615917,
2025-04-25T04:43:07.0286615Z 			"digest": "sha256:30970457a9f9083a2a5dae43c422817ca999069fea78e8539ffd595b9864e06d"
2025-04-25T04:43:07.0287435Z 		},
2025-04-25T04:43:07.0287791Z 		{
2025-04-25T04:43:07.0288395Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0289129Z 			"size": 161,
2025-04-25T04:43:07.0289847Z 			"digest": "sha256:df48990ef7192f7155a517f7b47d2dc3e20ff21ec70594dd13c8a72045d2d289"
2025-04-25T04:43:07.0290653Z 		},
2025-04-25T04:43:07.0290992Z 		{
2025-04-25T04:43:07.0291584Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0292399Z 			"size": 2020,
2025-04-25T04:43:07.0293375Z 			"digest": "sha256:a4cced2d068fe50e5b119ac2872768b377fc829d1b6d9aeada22e8452da0fdf5"
2025-04-25T04:43:07.0294231Z 		},
2025-04-25T04:43:07.0294574Z 		{
2025-04-25T04:43:07.0295160Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0295901Z 			"size": 8666608553,
2025-04-25T04:43:07.0296688Z 			"digest": "sha256:78c987ecaad25ddf7c7abe24802f773bbd6028d7dd9b2e86449e55dc5fe63a9b"
2025-04-25T04:43:07.0297540Z 		},
2025-04-25T04:43:07.0297874Z 		{
2025-04-25T04:43:07.0298466Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0299195Z 			"size": 105,
2025-04-25T04:43:07.0299927Z 			"digest": "sha256:5f8204dc9b54c2dab183625b10cac96fc24f7a372b84d8ea9e4d9c9c2c43701e"
2025-04-25T04:43:07.0300762Z 		},
2025-04-25T04:43:07.0301108Z 		{
2025-04-25T04:43:07.0301698Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0302513Z 			"size": 542,
2025-04-25T04:43:07.0303368Z 			"digest": "sha256:453718775e9b34916dddee754deb1e4bb11c44fdf5946bd56dc6cc51546c689d"
2025-04-25T04:43:07.0304249Z 		},
2025-04-25T04:43:07.0304597Z 		{
2025-04-25T04:43:07.0305181Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0305921Z 			"size": 682231098,
2025-04-25T04:43:07.0306666Z 			"digest": "sha256:21799f90e1a87dfeda4e01da87df5072b307bfa90e49bc869d7558e3802e06d4"
2025-04-25T04:43:07.0307501Z 		},
2025-04-25T04:43:07.0307841Z 		{
2025-04-25T04:43:07.0308426Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0309159Z 			"size": 111,
2025-04-25T04:43:07.0309891Z 			"digest": "sha256:a5fdc1fc436cc8a2d5bba8255bea8cead0b88a073728b8c9bbfcf441566b9ab8"
2025-04-25T04:43:07.0310748Z 		},
2025-04-25T04:43:07.0311096Z 		{
2025-04-25T04:43:07.0311676Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0312420Z 			"size": 1555,
2025-04-25T04:43:07.0313147Z 			"digest": "sha256:8a531b8941d085f579a0fd643b1d48a39c65a12a6619087ace6521d96e70e17f"
2025-04-25T04:43:07.0313967Z 		},
2025-04-25T04:43:07.0314310Z 		{
2025-04-25T04:43:07.0314885Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0315620Z 			"size": 107,
2025-04-25T04:43:07.0316344Z 			"digest": "sha256:9696eb6eaac88ebfcf4efb3b7ac82549ba5d75e1c90c4e1314a93d94629e4b25"
2025-04-25T04:43:07.0317210Z 		},
2025-04-25T04:43:07.0317564Z 		{
2025-04-25T04:43:07.0318144Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0318875Z 			"size": 166,
2025-04-25T04:43:07.0319720Z 			"digest": "sha256:875e6c05b7f451915266e05baa582245b7b4a01fcbdeec07b35f194d9fbb6914"
2025-04-25T04:43:07.0320576Z 		},
2025-04-25T04:43:07.0320932Z 		{
2025-04-25T04:43:07.0321594Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0322842Z 			"size": 2726472,
2025-04-25T04:43:07.0323696Z 			"digest": "sha256:c55088ba2941261ffcbfbf199c0b76410ef34cdd9314833e713e5f7091176be3"
2025-04-25T04:43:07.0324529Z 		},
2025-04-25T04:43:07.0324873Z 		{
2025-04-25T04:43:07.0325450Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0326184Z 			"size": 107,
2025-04-25T04:43:07.0326898Z 			"digest": "sha256:11d0f2e1766140314ad539a6bf8d20b5e9070493e79eb356f79bb330389b1bbd"
2025-04-25T04:43:07.0328039Z 		},
2025-04-25T04:43:07.0328382Z 		{
2025-04-25T04:43:07.0328953Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0329696Z 			"size": 801,
2025-04-25T04:43:07.0330423Z 			"digest": "sha256:ef8ec05d5147ccf77269b66fc4c20d61b8f709b8b524728e6efaa913e7b81ee9"
2025-04-25T04:43:07.0331256Z 		},
2025-04-25T04:43:07.0331599Z 		{
2025-04-25T04:43:07.0332187Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0332927Z 			"size": 26660309,
2025-04-25T04:43:07.0333697Z 			"digest": "sha256:d13c91bc646d4298276ced358d42ec4dec76d1a7ad5a54e647cd0f1ab7dfff21"
2025-04-25T04:43:07.0334535Z 		},
2025-04-25T04:43:07.0334867Z 		{
2025-04-25T04:43:07.0335450Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0336182Z 			"size": 104,
2025-04-25T04:43:07.0336908Z 			"digest": "sha256:f2dc05ea6d1c881149268ad6d5b9af4816e90bb0a90d9e7a37f8166eb1a0c910"
2025-04-25T04:43:07.0337736Z 		},
2025-04-25T04:43:07.0338095Z 		{
2025-04-25T04:43:07.0338673Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0339410Z 			"size": 425,
2025-04-25T04:43:07.0340184Z 			"digest": "sha256:24aecc8cff98df1a77820cbda635834dbae38fafad4bb0cdb9ee1bddeb59058e"
2025-04-25T04:43:07.0341046Z 		},
2025-04-25T04:43:07.0341379Z 		{
2025-04-25T04:43:07.0341953Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0342811Z 			"size": 19199347,
2025-04-25T04:43:07.0343700Z 			"digest": "sha256:5be8594d0dd596c18c9b5888c86dfdece20e604372c902be005340bb6444eb49"
2025-04-25T04:43:07.0344537Z 		},
2025-04-25T04:43:07.0344876Z 		{
2025-04-25T04:43:07.0345453Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0346184Z 			"size": 642,
2025-04-25T04:43:07.0346910Z 			"digest": "sha256:b97b5e4224f57fcdf563ee5492e2060a12dcce1af10c7bc62800a2ef029ecd56"
2025-04-25T04:43:07.0347729Z 		},
2025-04-25T04:43:07.0348069Z 		{
2025-04-25T04:43:07.0348646Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0349379Z 			"size": 701,
2025-04-25T04:43:07.0350111Z 			"digest": "sha256:7cdffbeea3da2f9187485c11a0b245cec525a103720990daf8ae5d9c9be089a4"
2025-04-25T04:43:07.0350952Z 		},
2025-04-25T04:43:07.0351290Z 		{
2025-04-25T04:43:07.0351872Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0352601Z 			"size": 141,
2025-04-25T04:43:07.0353320Z 			"digest": "sha256:7491c96ceff50f985bf0f05cfab1ea47b5005f5f1a763702d96d61e19af8bc99"
2025-04-25T04:43:07.0354172Z 		},
2025-04-25T04:43:07.0354518Z 		{
2025-04-25T04:43:07.0355082Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0355813Z 			"size": 135,
2025-04-25T04:43:07.0356541Z 			"digest": "sha256:101b3025b11e7f2f5a70e62f9e3cb67c9becd47ba56dafc6405ff5a463e6e146"
2025-04-25T04:43:07.0357374Z 		},
2025-04-25T04:43:07.0357710Z 		{
2025-04-25T04:43:07.0358284Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0359024Z 			"size": 5221737316,
2025-04-25T04:43:07.0360144Z 			"digest": "sha256:95285d7f27433946ce5154907b8baefb0da88e7852928c0f643743a1f5281afe"
2025-04-25T04:43:07.0360977Z 		},
2025-04-25T04:43:07.0361419Z 		{
2025-04-25T04:43:07.0362099Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0362977Z 			"size": 193,
2025-04-25T04:43:07.0363760Z 			"digest": "sha256:f6457eb37053418c22e0afbb6a309e890c235a07d84f52a52c2a4c8ac0feed9d"
2025-04-25T04:43:07.0364575Z 		},
2025-04-25T04:43:07.0365268Z 		{
2025-04-25T04:43:07.0365873Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0366530Z 			"size": 563,
2025-04-25T04:43:07.0367137Z 			"digest": "sha256:08d09dcfe8fc543f09a4980be8b1a9d4f34d51871ea671247d7a50fa02b69b20"
2025-04-25T04:43:07.0367837Z 		},
2025-04-25T04:43:07.0368137Z 		{
2025-04-25T04:43:07.0368626Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0369594Z 			"size": 43204573,
2025-04-25T04:43:07.0370234Z 			"digest": "sha256:809188ae2c9e9fc25746a808a46d76858a5598be1b2bb19bb878d18bf58e04d1"
2025-04-25T04:43:07.0370960Z 		},
2025-04-25T04:43:07.0371266Z 		{
2025-04-25T04:43:07.0371740Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0372350Z 			"size": 106,
2025-04-25T04:43:07.0372940Z 			"digest": "sha256:8774078b5a4c9ba0ea2a8c8f974be71977f25670c77c085906eb31362c217cd4"
2025-04-25T04:43:07.0373629Z 		},
2025-04-25T04:43:07.0373915Z 		{
2025-04-25T04:43:07.0374401Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0375016Z 			"size": 345,
2025-04-25T04:43:07.0375611Z 			"digest": "sha256:092e88537c972d5357b36bcd82a5cda6b9a59793733550bc0b10d5a13e85a27a"
2025-04-25T04:43:07.0376284Z 		},
2025-04-25T04:43:07.0376565Z 		{
2025-04-25T04:43:07.0377066Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0377692Z 			"size": 88292,
2025-04-25T04:43:07.0378353Z 			"digest": "sha256:76b8e2afd5da4f8c3900eb6afbd19fca3cb58e63f658d27bd0a0c1cebfb6124e"
2025-04-25T04:43:07.0379085Z 		},
2025-04-25T04:43:07.0379384Z 		{
2025-04-25T04:43:07.0379872Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0380504Z 			"size": 106,
2025-04-25T04:43:07.0381103Z 			"digest": "sha256:2632a791449064237077b2b75a9a7feaba63f29652ed7d63a6ec148a50f89e5f"
2025-04-25T04:43:07.0381788Z 		},
2025-04-25T04:43:07.0382069Z 		{
2025-04-25T04:43:07.0382653Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0383412Z 			"size": 1561,
2025-04-25T04:43:07.0384095Z 			"digest": "sha256:85e96d12bd1f07610d764eb39045abc44d38d3ee806a25ecb49c5ca4c4a415ea"
2025-04-25T04:43:07.0384784Z 		},
2025-04-25T04:43:07.0385071Z 		{
2025-04-25T04:43:07.0385554Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0386170Z 			"size": 701,
2025-04-25T04:43:07.0386800Z 			"digest": "sha256:7cdffbeea3da2f9187485c11a0b245cec525a103720990daf8ae5d9c9be089a4"
2025-04-25T04:43:07.0387510Z 		},
2025-04-25T04:43:07.0387808Z 		{
2025-04-25T04:43:07.0388287Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0388905Z 			"size": 139,
2025-04-25T04:43:07.0389516Z 			"digest": "sha256:305f5dc736d68e1a74f75eb45df189b23dedf2f35e0bff33382de1cf36a4a735"
2025-04-25T04:43:07.0390224Z 		},
2025-04-25T04:43:07.0390519Z 		{
2025-04-25T04:43:07.0390996Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0391624Z 			"size": 120,
2025-04-25T04:43:07.0392235Z 			"digest": "sha256:e6b720dd00f4debd19bf1b5f0195d9154ed69ec15d64fafa869344156738fe87"
2025-04-25T04:43:07.0392965Z 		},
2025-04-25T04:43:07.0393269Z 		{
2025-04-25T04:43:07.0393763Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0394392Z 			"size": 5737758455,
2025-04-25T04:43:07.0395046Z 			"digest": "sha256:c26d112ffc69c7cd4098bc6e8b21397ebadc11300ae7479c3e7501737604a64b"
2025-04-25T04:43:07.0395770Z 		},
2025-04-25T04:43:07.0396070Z 		{
2025-04-25T04:43:07.0396569Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0397186Z 			"size": 175,
2025-04-25T04:43:07.0397815Z 			"digest": "sha256:a0746c173a2d9be8f1c90ae54a38362e760aade84a45161bd4172f0b54c23991"
2025-04-25T04:43:07.0398514Z 		},
2025-04-25T04:43:07.0398807Z 		{
2025-04-25T04:43:07.0399298Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0400327Z 			"size": 1899,
2025-04-25T04:43:07.0400960Z 			"digest": "sha256:948a3301114ee69ba31ac7d858e9c7c7a7023d671901754ba0985ccc4071ee97"
2025-04-25T04:43:07.0401776Z 		},
2025-04-25T04:43:07.0402122Z 		{
2025-04-25T04:43:07.0402682Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0403413Z 			"size": 196432100,
2025-04-25T04:43:07.0404030Z 			"digest": "sha256:8a3074fe3294590d7104a606ca775300d5f46c309b41d9f40ba101463db71bae"
2025-04-25T04:43:07.0404986Z 		},
2025-04-25T04:43:07.0405274Z 		{
2025-04-25T04:43:07.0405765Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0406386Z 			"size": 300,
2025-04-25T04:43:07.0406993Z 			"digest": "sha256:4971ce4bca57334583f4851cd7408a713db284ffc6a220ebb3ee999c3bfb4715"
2025-04-25T04:43:07.0407690Z 		},
2025-04-25T04:43:07.0407984Z 		{
2025-04-25T04:43:07.0408458Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0409084Z 			"size": 32,
2025-04-25T04:43:07.0409717Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-04-25T04:43:07.0410426Z 		},
2025-04-25T04:43:07.0410717Z 		{
2025-04-25T04:43:07.0411202Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0411825Z 			"size": 108,
2025-04-25T04:43:07.0412448Z 			"digest": "sha256:0fb1d194c395c700a9430e1d0d8aeece61019de638dac9f94a65263a4f6480a6"
2025-04-25T04:43:07.0413157Z 		},
2025-04-25T04:43:07.0413459Z 		{
2025-04-25T04:43:07.0413952Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-25T04:43:07.0414581Z 			"size": 54145699,
2025-04-25T04:43:07.0415205Z 			"digest": "sha256:a8aeb18a88355ba432b516c79604dc852b3ed124d866f30145a416ec48e38986"
2025-04-25T04:43:07.0415906Z 		}
2025-04-25T04:43:07.0416201Z 	]
2025-04-25T04:43:07.0416513Z }
2025-04-25T04:43:07.0416849Z + exit 0
2025-04-25T04:43:07.0462067Z ##[group]Run set -eux
2025-04-25T04:43:07.0462549Z set -eux
2025-04-25T04:43:07.0464107Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2025-04-25T04:43:07.0515209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:07.0515825Z env:
2025-04-25T04:43:07.0516171Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:07.0516844Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:07.0517875Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:07.0518786Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:07.0520453Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:07.0521829Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:07.0522284Z   AWS_REGION: us-east-1
2025-04-25T04:43:07.0522832Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:07.0523419Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:07.0532731Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:07.0533141Z ##[endgroup]
2025-04-25T04:43:07.0627506Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2025-04-25T04:43:07.0630849Z + jq --raw-output .SecretString
2025-04-25T04:43:07.0636102Z + jq -r .docker_hub_readonly_token
2025-04-25T04:43:07.0640252Z + docker login --username pytorchbot --password-stdin
2025-04-25T04:43:07.7654071Z 
2025-04-25T04:43:07.7656875Z An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_s3_and_ecr_read_only/GitHubActions is not authorized to perform: secretsmanager:GetSecretValue on resource: docker_hub_readonly_token because no identity-based policy allows the secretsmanager:GetSecretValue action
2025-04-25T04:43:07.8554522Z Error: Cannot perform an interactive login from a non TTY device
2025-04-25T04:43:07.8630427Z ##[error]Process completed with exit code 1.
2025-04-25T04:43:07.8864575Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2025-04-25T04:43:07.8865297Z with:
2025-04-25T04:43:07.8866289Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:07.8867522Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:07.8868483Z env:
2025-04-25T04:43:07.8868828Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:07.8869492Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:07.8870449Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:07.8871348Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:07.8872912Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:07.8874271Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:07.8874731Z   AWS_REGION: us-east-1
2025-04-25T04:43:07.8875245Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:07.8875829Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:07.8885352Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:07.8885752Z ##[endgroup]
2025-04-25T04:43:07.8913340Z ##[group]Run set -x
2025-04-25T04:43:07.8913771Z set -x
2025-04-25T04:43:07.8914127Z set +e
2025-04-25T04:43:07.8914471Z 
2025-04-25T04:43:07.8914801Z login() {
2025-04-25T04:43:07.8915545Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-04-25T04:43:07.8916331Z }
2025-04-25T04:43:07.8916652Z 
2025-04-25T04:43:07.8916970Z retry () {
2025-04-25T04:43:07.8917392Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-04-25T04:43:07.8917874Z }
2025-04-25T04:43:07.8918185Z 
2025-04-25T04:43:07.8918559Z retry login "${DOCKER_REGISTRY}"
2025-04-25T04:43:07.8919037Z 
2025-04-25T04:43:07.8919910Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024')
2025-04-25T04:43:07.8920905Z echo "Compressed size of image in MB: ${IMAGE_SIZE}"
2025-04-25T04:43:07.8921501Z 
2025-04-25T04:43:07.8921824Z set -e
2025-04-25T04:43:07.8922363Z # ignore output since only exit code is used for conditional
2025-04-25T04:43:07.8923116Z # only pull docker image if it's not available locally
2025-04-25T04:43:07.8923936Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2025-04-25T04:43:07.8924689Z   retry docker pull "${DOCKER_IMAGE}"
2025-04-25T04:43:07.8925170Z fi
2025-04-25T04:43:07.8972621Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:07.8973336Z env:
2025-04-25T04:43:07.8973780Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:07.8974592Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:07.8975599Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:07.8976511Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:07.8978042Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:07.8979405Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:07.8979859Z   AWS_REGION: us-east-1
2025-04-25T04:43:07.8980438Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:07.8981138Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:07.8990778Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:07.8992239Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:07.8993479Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:07.8994094Z ##[endgroup]
2025-04-25T04:43:07.9082713Z + set +e
2025-04-25T04:43:07.9083347Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:07.9084152Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:07.9091910Z + aws ecr get-login-password --region us-east-1
2025-04-25T04:43:07.9095341Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T04:43:08.9877352Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-04-25T04:43:08.9878503Z Configure a credential helper to remove this warning. See
2025-04-25T04:43:08.9879681Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-04-25T04:43:08.9880412Z 
2025-04-25T04:43:08.9885558Z Login Succeeded
2025-04-25T04:43:08.9955724Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:08.9957311Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024'
2025-04-25T04:43:09.7274444Z + IMAGE_SIZE=23353.568830490112
2025-04-25T04:43:09.7275135Z Compressed size of image in MB: 23353.568830490112
2025-04-25T04:43:09.7275920Z + echo 'Compressed size of image in MB: 23353.568830490112'
2025-04-25T04:43:09.7276643Z + set -e
2025-04-25T04:43:09.7277884Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:09.7818499Z Prepare all required actions
2025-04-25T04:43:09.7819358Z Getting action download info
2025-04-25T04:43:09.9332819Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2025-04-25T04:43:10.5016381Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093)
2025-04-25T04:43:11.0306351Z ##[group]Run ./.github/actions/download-build-artifacts
2025-04-25T04:43:11.0306862Z with:
2025-04-25T04:43:11.0307200Z   name: linux-focal-rocm-py3.10
2025-04-25T04:43:11.0307633Z   s3-bucket: gha-artifacts
2025-04-25T04:43:11.0307998Z env:
2025-04-25T04:43:11.0308315Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:11.0308914Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:11.0309818Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:11.0310683Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:11.0312044Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:11.0313252Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:11.0313651Z   AWS_REGION: us-east-1
2025-04-25T04:43:11.0314140Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:11.0314685Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:11.0323404Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:11.0323840Z ##[endgroup]
2025-04-25T04:43:11.0367681Z ##[group]Run seemethere/download-artifact-s3@v4
2025-04-25T04:43:11.0368152Z with:
2025-04-25T04:43:11.0368472Z   name: linux-focal-rocm-py3.10
2025-04-25T04:43:11.0368902Z   s3-bucket: gha-artifacts
2025-04-25T04:43:11.0369276Z   region: us-east-1
2025-04-25T04:43:11.0369594Z env:
2025-04-25T04:43:11.0369890Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:11.0370482Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:11.0371345Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:11.0372145Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:11.0373506Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:11.0374711Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:11.0375112Z   AWS_REGION: us-east-1
2025-04-25T04:43:11.0375553Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:11.0376081Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:11.0384838Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:11.0385562Z ##[endgroup]
2025-04-25T04:43:11.5004823Z (node:2384621) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-04-25T04:43:11.5005773Z 
2025-04-25T04:43:11.5006188Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-04-25T04:43:11.5007160Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-04-25T04:43:11.5008184Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-04-25T04:43:11.6602904Z Found 1 objects with prefix pytorch/pytorch/14656749847/linux-focal-rocm-py3.10/
2025-04-25T04:43:11.6604259Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-04-25T04:43:21.3335573Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-04-25T04:43:21.3346951Z Artifact download has finished successfully
2025-04-25T04:43:21.4024044Z ##[group]Run unzip -o artifacts.zip
2025-04-25T04:43:21.4024794Z unzip -o artifacts.zip
2025-04-25T04:43:21.4082910Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:21.4083653Z env:
2025-04-25T04:43:21.4084079Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:21.4085468Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:21.4086498Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:21.4087409Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:21.4089002Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:21.4090387Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:21.4090833Z   AWS_REGION: us-east-1
2025-04-25T04:43:21.4091518Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:21.4092131Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:21.4101480Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:21.4101892Z ##[endgroup]
2025-04-25T04:43:21.4229968Z Archive:  artifacts.zip
2025-04-25T04:43:21.4230542Z    creating: dist/
2025-04-25T04:43:24.8862745Z   inflating: dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl  
2025-04-25T04:43:24.8989740Z   inflating: dist/.ninja_log         
2025-04-25T04:43:24.8990446Z    creating: build/custom_test_artifacts/
2025-04-25T04:43:24.8991189Z    creating: build/custom_test_artifacts/custom-op-build/
2025-04-25T04:43:24.8992072Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2025-04-25T04:43:24.8993103Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2025-04-25T04:43:24.8994761Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-04-25T04:43:24.8995953Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/
2025-04-25T04:43:24.8997073Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeSystem.cmake  
2025-04-25T04:43:24.8998324Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/
2025-04-25T04:43:24.8999681Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/tmp/
2025-04-25T04:43:24.9001028Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c  
2025-04-25T04:43:24.9002427Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/a.out  
2025-04-25T04:43:24.9003752Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake  
2025-04-25T04:43:24.9005001Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/
2025-04-25T04:43:24.9006199Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/
2025-04-25T04:43:24.9007643Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-04-25T04:43:24.9009826Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out  
2025-04-25T04:43:24.9011140Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake  
2025-04-25T04:43:24.9012594Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin  
2025-04-25T04:43:24.9014123Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin  
2025-04-25T04:43:24.9015439Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2025-04-25T04:43:24.9016481Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2025-04-25T04:43:24.9017579Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2025-04-25T04:43:24.9018721Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2025-04-25T04:43:24.9019985Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2025-04-25T04:43:24.9021416Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2025-04-25T04:43:24.9023193Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2025-04-25T04:43:24.9024513Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2025-04-25T04:43:24.9025820Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2025-04-25T04:43:24.9027131Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2025-04-25T04:43:24.9028441Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2025-04-25T04:43:24.9029745Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2025-04-25T04:43:24.9031048Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2025-04-25T04:43:24.9036776Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2025-04-25T04:43:24.9219632Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2025-04-25T04:43:24.9220998Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2025-04-25T04:43:24.9222391Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2025-04-25T04:43:24.9223907Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2025-04-25T04:43:24.9225335Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2025-04-25T04:43:24.9226689Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2025-04-25T04:43:24.9268610Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2025-04-25T04:43:24.9270193Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2025-04-25T04:43:24.9271699Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2025-04-25T04:43:24.9273231Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2025-04-25T04:43:24.9274681Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2025-04-25T04:43:24.9276208Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2025-04-25T04:43:24.9323383Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2025-04-25T04:43:24.9325042Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-04-25T04:43:24.9326969Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2025-04-25T04:43:24.9328254Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2025-04-25T04:43:24.9329393Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2025-04-25T04:43:24.9330496Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2025-04-25T04:43:24.9331806Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_vec_ext.cc  
2025-04-25T04:43:24.9333013Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2025-04-25T04:43:24.9333977Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2025-04-25T04:43:24.9334978Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2025-04-25T04:43:24.9479380Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2025-04-25T04:43:24.9529836Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2025-04-25T04:43:24.9530874Z    creating: build/custom_test_artifacts/jit-hook-build/
2025-04-25T04:43:24.9532224Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2025-04-25T04:43:24.9533316Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2025-04-25T04:43:24.9534551Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-04-25T04:43:24.9535678Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/
2025-04-25T04:43:24.9536796Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeSystem.cmake  
2025-04-25T04:43:24.9538011Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/
2025-04-25T04:43:24.9539197Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/tmp/
2025-04-25T04:43:24.9540543Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c  
2025-04-25T04:43:24.9541914Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/a.out  
2025-04-25T04:43:24.9543195Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake  
2025-04-25T04:43:24.9544504Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/
2025-04-25T04:43:24.9545683Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/
2025-04-25T04:43:24.9547084Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-04-25T04:43:24.9548516Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out  
2025-04-25T04:43:24.9549824Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake  
2025-04-25T04:43:24.9551252Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin  
2025-04-25T04:43:24.9552788Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin  
2025-04-25T04:43:24.9554084Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2025-04-25T04:43:24.9555113Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2025-04-25T04:43:24.9556187Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2025-04-25T04:43:24.9557328Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2025-04-25T04:43:24.9558646Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2025-04-25T04:43:24.9560519Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2025-04-25T04:43:24.9561938Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2025-04-25T04:43:24.9563624Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2025-04-25T04:43:24.9564987Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2025-04-25T04:43:24.9566352Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2025-04-25T04:43:24.9567718Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2025-04-25T04:43:24.9569075Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2025-04-25T04:43:24.9570401Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2025-04-25T04:43:24.9573878Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2025-04-25T04:43:24.9633224Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2025-04-25T04:43:24.9634700Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-04-25T04:43:24.9636487Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2025-04-25T04:43:24.9637746Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2025-04-25T04:43:24.9638887Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2025-04-25T04:43:24.9640314Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2025-04-25T04:43:24.9641423Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_vec_ext.cc  
2025-04-25T04:43:24.9642452Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2025-04-25T04:43:24.9643412Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2025-04-25T04:43:24.9644400Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2025-04-25T04:43:24.9674525Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2025-04-25T04:43:24.9675503Z    creating: build/custom_test_artifacts/custom-backend-build/
2025-04-25T04:43:24.9676435Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2025-04-25T04:43:24.9677529Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2025-04-25T04:43:24.9679250Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-04-25T04:43:24.9680605Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/
2025-04-25T04:43:24.9681801Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeSystem.cmake  
2025-04-25T04:43:24.9683101Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/
2025-04-25T04:43:24.9684389Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/tmp/
2025-04-25T04:43:24.9685825Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c  
2025-04-25T04:43:24.9687287Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/a.out  
2025-04-25T04:43:24.9688662Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake  
2025-04-25T04:43:24.9689988Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/
2025-04-25T04:43:24.9691269Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/
2025-04-25T04:43:24.9692763Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-04-25T04:43:24.9694307Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out  
2025-04-25T04:43:24.9696160Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake  
2025-04-25T04:43:24.9697676Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin  
2025-04-25T04:43:24.9699295Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin  
2025-04-25T04:43:24.9700682Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2025-04-25T04:43:24.9701792Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2025-04-25T04:43:24.9702961Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2025-04-25T04:43:24.9704193Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2025-04-25T04:43:24.9705599Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2025-04-25T04:43:24.9707181Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2025-04-25T04:43:24.9709009Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2025-04-25T04:43:24.9710434Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2025-04-25T04:43:24.9711913Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2025-04-25T04:43:24.9713422Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2025-04-25T04:43:24.9714916Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2025-04-25T04:43:24.9716410Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2025-04-25T04:43:24.9717868Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2025-04-25T04:43:24.9719441Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2025-04-25T04:43:24.9818080Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2025-04-25T04:43:24.9819620Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2025-04-25T04:43:24.9821137Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2025-04-25T04:43:24.9822834Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2025-04-25T04:43:24.9824445Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2025-04-25T04:43:24.9825992Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2025-04-25T04:43:24.9827569Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2025-04-25T04:43:24.9829135Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2025-04-25T04:43:24.9830713Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2025-04-25T04:43:24.9832284Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2025-04-25T04:43:24.9833836Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2025-04-25T04:43:24.9843050Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2025-04-25T04:43:24.9894379Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2025-04-25T04:43:24.9896463Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-04-25T04:43:24.9897904Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2025-04-25T04:43:24.9899246Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2025-04-25T04:43:24.9900476Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2025-04-25T04:43:24.9901671Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2025-04-25T04:43:24.9902864Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_vec_ext.cc  
2025-04-25T04:43:24.9903992Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2025-04-25T04:43:24.9905029Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2025-04-25T04:43:24.9906094Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2025-04-25T04:43:24.9994289Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2025-04-25T04:43:25.0030196Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2025-04-25T04:43:25.0031079Z    creating: build/lib/
2025-04-25T04:43:25.0107151Z   inflating: build/lib/libprotobuf-lite.a  
2025-04-25T04:43:25.0517892Z   inflating: build/lib/libprotobuf.a  
2025-04-25T04:43:25.0974896Z   inflating: build/lib/libprotoc.a   
2025-04-25T04:43:25.0983314Z   inflating: build/lib/libpthreadpool.a  
2025-04-25T04:43:25.0990923Z   inflating: build/lib/libcpuinfo.a  
2025-04-25T04:43:25.0998173Z   inflating: build/lib/libcpuinfo_internals.a  
2025-04-25T04:43:25.0998927Z   inflating: build/lib/libclog.a     
2025-04-25T04:43:25.1017893Z   inflating: build/lib/libpytorch_qnnpack.a  
2025-04-25T04:43:25.1018754Z   inflating: build/lib/libnnpack_reference_layers.a  
2025-04-25T04:43:25.1035879Z   inflating: build/lib/libnnpack.a   
2025-04-25T04:43:25.1211037Z   inflating: build/lib/libmicrokernels-prod.a  
2025-04-25T04:43:25.2025784Z   inflating: build/lib/libmicrokernels-all.a  
2025-04-25T04:43:25.2089528Z   inflating: build/lib/libgtest.a    
2025-04-25T04:43:25.2105927Z   inflating: build/lib/libgmock.a    
2025-04-25T04:43:25.2106641Z   inflating: build/lib/libgmock_main.a  
2025-04-25T04:43:25.2107285Z   inflating: build/lib/libgtest_main.a  
2025-04-25T04:43:25.2190087Z   inflating: build/lib/libXNNPACK.a  
2025-04-25T04:43:25.2257205Z   inflating: build/lib/libbenchmark.a  
2025-04-25T04:43:25.2257947Z   inflating: build/lib/libbenchmark_main.a  
2025-04-25T04:43:25.2264682Z   inflating: build/lib/libittnotify.a  
2025-04-25T04:43:25.2322492Z   inflating: build/lib/libasmjit.a   
2025-04-25T04:43:25.3313969Z   inflating: build/lib/libfbgemm.a   
2025-04-25T04:43:25.3338681Z   inflating: build/lib/libtensorpipe_uv.a  
2025-04-25T04:43:25.3861280Z   inflating: build/lib/libtensorpipe.a  
2025-04-25T04:43:25.3974361Z   inflating: build/lib/libgloo.a     
2025-04-25T04:43:25.4012007Z   inflating: build/lib/libonnx_proto.a  
2025-04-25T04:43:25.4371578Z   inflating: build/lib/libgloo_hip.a  
2025-04-25T04:43:25.5046871Z   inflating: build/lib/libonnx.a     
2025-04-25T04:43:26.4659021Z   inflating: build/lib/libdnnl.a     
2025-04-25T04:43:26.4675713Z   inflating: build/lib/libfmt.a      
2025-04-25T04:43:26.4940867Z   inflating: build/lib/libkineto.a   
2025-04-25T04:43:26.5036814Z   inflating: build/lib/libc10.so     
2025-04-25T04:43:26.5037556Z   inflating: build/lib/libtorch_global_deps.so  
2025-04-25T04:43:26.5038288Z   inflating: build/lib/libcaffe2_nvrtc.so  
2025-04-25T04:43:26.5084552Z   inflating: build/lib/libc10_hip.so  
2025-04-25T04:43:28.7361923Z   inflating: build/lib/libtorch_cpu.so  
2025-04-25T04:43:28.7365921Z   inflating: build/lib/libunbox_lib.a  
2025-04-25T04:43:28.7369563Z   inflating: build/lib/libshm.so     
2025-04-25T04:43:29.3491968Z   inflating: build/lib/libtorch_hip.so  
2025-04-25T04:43:29.3493347Z   inflating: build/lib/libtorch.so   
2025-04-25T04:43:29.3509803Z   inflating: build/lib/libjitbackend_test.so  
2025-04-25T04:43:29.3531528Z   inflating: build/lib/libbackend_with_compiler.so  
2025-04-25T04:43:29.3594849Z   inflating: build/lib/libtorchbind_test.so  
2025-04-25T04:43:29.3616463Z   inflating: build/lib/libaoti_custom_ops.so  
2025-04-25T04:43:29.5412144Z   inflating: build/lib/libtorch_python.so  
2025-04-25T04:43:29.5442201Z   inflating: build/lib/libnnapi_backend.so  
2025-04-25T04:43:29.5442894Z    creating: build/bin/
2025-04-25T04:43:29.5443403Z    creating: build/bin/CMakeFiles/
2025-04-25T04:43:29.5443984Z   inflating: build/bin/cmake_install.cmake  
2025-04-25T04:43:29.5444593Z   inflating: build/bin/CTestTestfile.cmake  
2025-04-25T04:43:29.5852084Z   inflating: build/bin/protoc-3.13.0.0  
2025-04-25T04:43:29.6259729Z   inflating: build/bin/protoc        
2025-04-25T04:43:29.6309015Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2025-04-25T04:43:29.6360135Z   inflating: build/bin/c10_DeviceGuard_test  
2025-04-25T04:43:29.6410953Z   inflating: build/bin/c10_Device_test  
2025-04-25T04:43:29.6469237Z   inflating: build/bin/c10_DispatchKeySet_test  
2025-04-25T04:43:29.6523604Z   inflating: build/bin/c10_Scalar_test  
2025-04-25T04:43:29.6571759Z   inflating: build/bin/c10_StreamGuard_test  
2025-04-25T04:43:29.6625151Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2025-04-25T04:43:29.6675591Z   inflating: build/bin/c10_SymInt_test  
2025-04-25T04:43:29.6730336Z   inflating: build/bin/c10_InlineStreamGuard_test  
2025-04-25T04:43:29.6778968Z   inflating: build/bin/c10_ConstexprCrc_test  
2025-04-25T04:43:29.6833931Z   inflating: build/bin/c10_SizesAndStrides_test  
2025-04-25T04:43:29.6883240Z   inflating: build/bin/c10_ArrayRef_test  
2025-04-25T04:43:29.6951225Z   inflating: build/bin/c10_cow_test  
2025-04-25T04:43:29.7003847Z   inflating: build/bin/c10_Bitset_test  
2025-04-25T04:43:29.7053726Z   inflating: build/bin/c10_Half_test  
2025-04-25T04:43:29.7108982Z   inflating: build/bin/c10_LeftRight_test  
2025-04-25T04:43:29.7157984Z   inflating: build/bin/c10_DeadlockDetection_test  
2025-04-25T04:43:29.7210585Z   inflating: build/bin/c10_NetworkFlow_test  
2025-04-25T04:43:29.7259795Z   inflating: build/bin/c10_Synchronized_test  
2025-04-25T04:43:29.7314698Z   inflating: build/bin/c10_Metaprogramming_test  
2025-04-25T04:43:29.7369293Z   inflating: build/bin/c10_ThreadLocal_test  
2025-04-25T04:43:29.7420430Z   inflating: build/bin/c10_accumulate_test  
2025-04-25T04:43:29.7470670Z   inflating: build/bin/c10_TypeList_test  
2025-04-25T04:43:29.7521870Z   inflating: build/bin/c10_TypeIndex_test  
2025-04-25T04:43:29.7570514Z   inflating: build/bin/c10_TypeTraits_test  
2025-04-25T04:43:29.7625010Z   inflating: build/bin/c10_bfloat16_test  
2025-04-25T04:43:29.7674647Z   inflating: build/bin/c10_bit_cast_test  
2025-04-25T04:43:29.7723730Z   inflating: build/bin/c10_error_test  
2025-04-25T04:43:29.7776074Z   inflating: build/bin/c10_exception_test  
2025-04-25T04:43:29.7831780Z   inflating: build/bin/c10_complex_math_test  
2025-04-25T04:43:29.7885971Z   inflating: build/bin/c10_complex_test  
2025-04-25T04:43:29.7935814Z   inflating: build/bin/c10_flags_test  
2025-04-25T04:43:29.7988298Z   inflating: build/bin/c10_lazy_test  
2025-04-25T04:43:29.8037967Z   inflating: build/bin/c10_generic_math_test  
2025-04-25T04:43:29.8088755Z   inflating: build/bin/c10_irange_test  
2025-04-25T04:43:29.8246488Z   inflating: build/bin/c10_intrusive_ptr_test  
2025-04-25T04:43:29.8302422Z   inflating: build/bin/c10_logging_test  
2025-04-25T04:43:29.8375875Z   inflating: build/bin/c10_optional_test  
2025-04-25T04:43:29.8428241Z   inflating: build/bin/c10_registry_test  
2025-04-25T04:43:29.8488901Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2025-04-25T04:43:29.8637519Z   inflating: build/bin/c10_small_vector_test  
2025-04-25T04:43:29.8688416Z   inflating: build/bin/c10_ssize_test  
2025-04-25T04:43:29.8739731Z   inflating: build/bin/c10_string_util_test  
2025-04-25T04:43:29.8788274Z   inflating: build/bin/c10_string_view_test  
2025-04-25T04:43:29.8837980Z   inflating: build/bin/c10_tempfile_test  
2025-04-25T04:43:29.8893777Z   inflating: build/bin/c10_typeid_test  
2025-04-25T04:43:29.8937586Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2025-04-25T04:43:29.8986132Z   inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test  
2025-04-25T04:43:29.9034850Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream  
2025-04-25T04:43:29.9083446Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device  
2025-04-25T04:43:29.9132074Z   inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes  
2025-04-25T04:43:29.9180610Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads  
2025-04-25T04:43:29.9229317Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks  
2025-04-25T04:43:29.9277614Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block  
2025-04-25T04:43:29.9326208Z   inflating: build/bin/c10_hip_HIPTest  
2025-04-25T04:43:29.9720669Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2025-04-25T04:43:30.0124800Z   inflating: build/bin/vec_test_all_types_AVX512  
2025-04-25T04:43:30.0538797Z   inflating: build/bin/vec_test_all_types_AVX2  
2025-04-25T04:43:30.0590932Z   inflating: build/bin/test_edge_op_registration  
2025-04-25T04:43:30.0642756Z   inflating: build/bin/BackoffTest   
2025-04-25T04:43:30.0695004Z   inflating: build/bin/FileStoreTest  
2025-04-25T04:43:30.0751059Z   inflating: build/bin/TCPStoreTest  
2025-04-25T04:43:30.0803940Z   inflating: build/bin/HashStoreTest  
2025-04-25T04:43:30.0868329Z   inflating: build/bin/ProcessGroupGlooTest  
2025-04-25T04:43:30.0869911Z   inflating: build/bin/example_allreduce  
2025-04-25T04:43:30.0874569Z   inflating: build/bin/torch_shm_manager  
2025-04-25T04:43:30.0927758Z   inflating: build/bin/static_runtime_bench  
2025-04-25T04:43:30.1167037Z   inflating: build/bin/static_runtime_test  
2025-04-25T04:43:30.1238860Z   inflating: build/bin/Dict_test     
2025-04-25T04:43:30.1290815Z   inflating: build/bin/Dimname_test  
2025-04-25T04:43:30.1354664Z   inflating: build/bin/MaybeOwned_test  
2025-04-25T04:43:30.1410719Z   inflating: build/bin/NamedTensor_test  
2025-04-25T04:43:30.1468339Z   inflating: build/bin/apply_utils_test  
2025-04-25T04:43:30.1526010Z   inflating: build/bin/atest         
2025-04-25T04:43:30.1587633Z   inflating: build/bin/basic         
2025-04-25T04:43:30.1641744Z   inflating: build/bin/broadcast_test  
2025-04-25T04:43:30.1692050Z   inflating: build/bin/cpu_allocator_test  
2025-04-25T04:43:30.1749468Z   inflating: build/bin/cpu_generator_test  
2025-04-25T04:43:30.1801403Z   inflating: build/bin/cpu_profiling_allocator_test  
2025-04-25T04:43:30.1889645Z   inflating: build/bin/cpu_rng_test  
2025-04-25T04:43:30.1939315Z   inflating: build/bin/dlconvertor_test  
2025-04-25T04:43:30.1995589Z   inflating: build/bin/extension_backend_test  
2025-04-25T04:43:30.2049912Z   inflating: build/bin/half_test     
2025-04-25T04:43:30.2142240Z   inflating: build/bin/ivalue_test   
2025-04-25T04:43:30.2191478Z   inflating: build/bin/lazy_tensor_test  
2025-04-25T04:43:30.2244800Z   inflating: build/bin/math_kernel_test  
2025-04-25T04:43:30.2297316Z   inflating: build/bin/memory_format_test  
2025-04-25T04:43:30.2349774Z   inflating: build/bin/memory_overlapping_test  
2025-04-25T04:43:30.2400046Z   inflating: build/bin/operator_name_test  
2025-04-25T04:43:30.2452393Z   inflating: build/bin/mobile_memory_cleanup  
2025-04-25T04:43:30.2507250Z   inflating: build/bin/native_test   
2025-04-25T04:43:30.2557210Z   inflating: build/bin/operators_test  
2025-04-25T04:43:30.2608933Z   inflating: build/bin/packedtensoraccessor_test  
2025-04-25T04:43:30.2673773Z   inflating: build/bin/pow_test      
2025-04-25T04:43:30.2730136Z   inflating: build/bin/quantized_test  
2025-04-25T04:43:30.2779469Z   inflating: build/bin/reduce_ops_test  
2025-04-25T04:43:30.2835236Z   inflating: build/bin/scalar_tensor_test  
2025-04-25T04:43:30.2885578Z   inflating: build/bin/reportMemoryUsage_test  
2025-04-25T04:43:30.2936331Z   inflating: build/bin/StorageUtils_test  
2025-04-25T04:43:30.2993301Z   inflating: build/bin/scalar_test   
2025-04-25T04:43:30.3044691Z   inflating: build/bin/stride_properties_test  
2025-04-25T04:43:30.3099087Z   inflating: build/bin/type_ptr_test  
2025-04-25T04:43:30.3176011Z   inflating: build/bin/tensor_iterator_test  
2025-04-25T04:43:30.3177317Z   inflating: build/bin/thread_init_test  
2025-04-25T04:43:30.3231984Z   inflating: build/bin/test_parallel  
2025-04-25T04:43:30.3290183Z   inflating: build/bin/type_test     
2025-04-25T04:43:30.3341789Z   inflating: build/bin/undefined_tensor_test  
2025-04-25T04:43:30.3342545Z   inflating: build/bin/verify_api_visibility  
2025-04-25T04:43:30.3410915Z   inflating: build/bin/legacy_vmap_test  
2025-04-25T04:43:30.3461366Z   inflating: build/bin/weakref_test  
2025-04-25T04:43:30.3513015Z   inflating: build/bin/wrapdim_test  
2025-04-25T04:43:30.3615582Z   inflating: build/bin/List_test     
2025-04-25T04:43:30.3673792Z   inflating: build/bin/IListRef_test  
2025-04-25T04:43:30.3724965Z   inflating: build/bin/xla_tensor_test  
2025-04-25T04:43:30.3840495Z   inflating: build/bin/kernel_function_legacy_test  
2025-04-25T04:43:30.3931580Z   inflating: build/bin/kernel_function_test  
2025-04-25T04:43:30.3996022Z   inflating: build/bin/KernelFunction_test  
2025-04-25T04:43:30.4116542Z   inflating: build/bin/kernel_lambda_legacy_test  
2025-04-25T04:43:30.4175565Z   inflating: build/bin/kernel_stackbased_test  
2025-04-25T04:43:30.4273582Z   inflating: build/bin/kernel_lambda_test  
2025-04-25T04:43:30.4324122Z   inflating: build/bin/CppSignature_test  
2025-04-25T04:43:30.4415764Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2025-04-25T04:43:30.4464150Z   inflating: build/bin/op_allowlist_test  
2025-04-25T04:43:30.4518527Z   inflating: build/bin/backend_fallback_test  
2025-04-25T04:43:30.4583431Z   inflating: build/bin/inline_container_test  
2025-04-25T04:43:30.4632836Z   inflating: build/bin/hip_complex_math_test  
2025-04-25T04:43:30.4928093Z   inflating: build/bin/op_registration_test  
2025-04-25T04:43:30.4980059Z   inflating: build/bin/hip_apply_test  
2025-04-25T04:43:30.5028782Z   inflating: build/bin/hip_complex_test  
2025-04-25T04:43:30.5077409Z   inflating: build/bin/hip_distributions_test  
2025-04-25T04:43:30.5126036Z   inflating: build/bin/hip_generator_test  
2025-04-25T04:43:30.5174799Z   inflating: build/bin/hip_half_test  
2025-04-25T04:43:30.5223312Z   inflating: build/bin/hip_integer_divider_test  
2025-04-25T04:43:30.5271920Z   inflating: build/bin/hip_optional_test  
2025-04-25T04:43:30.5321206Z   inflating: build/bin/hip_packedtensoraccessor_test  
2025-04-25T04:43:30.5371576Z   inflating: build/bin/hip_dlconvertor_test  
2025-04-25T04:43:30.5420268Z   inflating: build/bin/hip_vectorized_test  
2025-04-25T04:43:30.5950090Z   inflating: build/bin/test_jit      
2025-04-25T04:43:30.6686543Z   inflating: build/bin/test_tensorexpr  
2025-04-25T04:43:30.6698743Z   inflating: build/bin/tutorial_tensorexpr  
2025-04-25T04:43:30.6752593Z   inflating: build/bin/test_dist_autograd  
2025-04-25T04:43:30.6818653Z   inflating: build/bin/test_cpp_rpc  
2025-04-25T04:43:30.7887037Z   inflating: build/bin/test_api      
2025-04-25T04:43:30.7888412Z   inflating: build/bin/parallel_benchmark  
2025-04-25T04:43:30.7953809Z   inflating: build/bin/test_mobile_nnc  
2025-04-25T04:43:30.7962703Z   inflating: build/bin/aot_model_compiler_test  
2025-04-25T04:43:30.8288450Z   inflating: build/bin/test_lazy     
2025-04-25T04:43:30.8289121Z    creating: .additional_ci_files/
2025-04-25T04:43:30.8390767Z   inflating: .additional_ci_files/test-times.json  
2025-04-25T04:43:30.8785395Z   inflating: .additional_ci_files/test-class-times.json  
2025-04-25T04:43:30.8846023Z ##[group]Run rm artifacts.zip
2025-04-25T04:43:30.8846762Z rm artifacts.zip
2025-04-25T04:43:30.8908253Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:30.8908992Z env:
2025-04-25T04:43:30.8909409Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:30.8910256Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:30.8911434Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:30.8912520Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:30.8914323Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:30.8915943Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:30.8916473Z   AWS_REGION: us-east-1
2025-04-25T04:43:30.8917105Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:30.8917818Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:30.8930104Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:30.8930708Z ##[endgroup]
2025-04-25T04:43:31.1366822Z ##[group]Run df -H
2025-04-25T04:43:31.1367281Z df -H
2025-04-25T04:43:31.1425196Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:31.1425911Z env:
2025-04-25T04:43:31.1426769Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:31.1427570Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:31.1428703Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:31.1429772Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:31.1431506Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:31.1433036Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:31.1433536Z   AWS_REGION: us-east-1
2025-04-25T04:43:31.1434126Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:31.1434830Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:31.1445733Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:31.1446200Z ##[endgroup]
2025-04-25T04:43:31.1601350Z Filesystem                                          Size  Used Avail Use% Mounted on
2025-04-25T04:43:31.1602450Z tmpfs                                               109G   33M  109G   1% /run
2025-04-25T04:43:31.1605391Z /dev/nvme0n1p2                                      944G   62G  835G   7% /
2025-04-25T04:43:31.1606259Z tmpfs                                               542G   33k  542G   1% /dev/shm
2025-04-25T04:43:31.1606992Z tmpfs                                               5.3M     0  5.3M   0% /run/lock
2025-04-25T04:43:31.1607747Z /dev/nvme0n1p1                                      536M  6.4M  530M   2% /boot/efi
2025-04-25T04:43:31.1608589Z /dev/nvme1n1p1                                      3.8T  702G  2.9T  20% /media/4TB
2025-04-25T04:43:31.1609426Z tmpfs                                               109G   33k  109G   1% /run/user/1307800118
2025-04-25T04:43:31.1610315Z 172.18.148.8:/export/amd2                           5.5T  260G  5.3T   5% /mnt
2025-04-25T04:43:31.1611371Z pure1.jax.cs.cpe.ice.amd.com:/homes/okakarpa        108G  587M  107G   1% /home/okakarpa
2025-04-25T04:43:31.1612707Z pure1.jax.cs.cpe.ice.amd.com:/homes/amd-pytorch     108G  1.9G  106G   2% /home/amd-pytorch
2025-04-25T04:43:31.1613836Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage          132T  119T   14T  90% /groups
2025-04-25T04:43:31.1614866Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage/Scratch  5.5T  1.8T  3.8T  32% /scratch
2025-04-25T04:43:31.1681186Z Prepare all required actions
2025-04-25T04:43:31.1681971Z Getting action download info
2025-04-25T04:43:31.3345180Z ##[group]Run ./.github/actions/download-td-artifacts
2025-04-25T04:43:31.3345808Z with:
2025-04-25T04:43:31.3346180Z env:
2025-04-25T04:43:31.3346579Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:31.3347368Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:31.3349013Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:31.3350120Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:31.3351981Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:31.3353625Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:31.3354158Z   AWS_REGION: us-east-1
2025-04-25T04:43:31.3354813Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:31.3355514Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:31.3366814Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:31.3367287Z ##[endgroup]
2025-04-25T04:43:31.3445004Z ##[group]Run seemethere/download-artifact-s3@v4
2025-04-25T04:43:31.3445620Z with:
2025-04-25T04:43:31.3445963Z   name: td_results
2025-04-25T04:43:31.3446353Z   s3-bucket: gha-artifacts
2025-04-25T04:43:31.3446816Z   region: us-east-1
2025-04-25T04:43:31.3447184Z env:
2025-04-25T04:43:31.3447525Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:31.3448195Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:31.3449232Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:31.3450178Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:31.3451751Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:31.3453163Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:31.3453626Z   AWS_REGION: us-east-1
2025-04-25T04:43:31.3454129Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:31.3454717Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:31.3464087Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:31.3464501Z ##[endgroup]
2025-04-25T04:43:31.8069199Z (node:2384657) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-04-25T04:43:31.8070146Z 
2025-04-25T04:43:31.8070503Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-04-25T04:43:31.8071464Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-04-25T04:43:31.8072447Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-04-25T04:43:31.9772580Z Found 1 objects with prefix pytorch/pytorch/14656749847/td_results/
2025-04-25T04:43:31.9773861Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2025-04-25T04:43:32.1645792Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2025-04-25T04:43:32.1656860Z Artifact download has finished successfully
2025-04-25T04:43:32.2285760Z ##[group]Run mkdir -p .additional_ci_files
2025-04-25T04:43:32.2286430Z mkdir -p .additional_ci_files
2025-04-25T04:43:32.2287225Z mv td_results.json .additional_ci_files/td_results.json || true
2025-04-25T04:43:32.2336364Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:32.2336730Z env:
2025-04-25T04:43:32.2336941Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:32.2337334Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:32.2337940Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:32.2338497Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:32.2339668Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:32.2340910Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:32.2341515Z   AWS_REGION: us-east-1
2025-04-25T04:43:32.2342205Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:32.2342987Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:32.2353891Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:32.2354364Z ##[endgroup]
2025-04-25T04:43:32.2567895Z ##[group]Run .github/scripts/parse_ref.py
2025-04-25T04:43:32.2568589Z .github/scripts/parse_ref.py
2025-04-25T04:43:32.2611592Z shell: /usr/bin/bash -e {0}
2025-04-25T04:43:32.2611864Z env:
2025-04-25T04:43:32.2612084Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:32.2612487Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:32.2613075Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:32.2613642Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:32.2614542Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:32.2615340Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:32.2615626Z   AWS_REGION: us-east-1
2025-04-25T04:43:32.2615915Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:32.2616290Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:32.2624190Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:32.2624649Z ##[endgroup]
2025-04-25T04:43:32.3173814Z Prepare all required actions
2025-04-25T04:43:32.3229210Z ##[group]Run ./.github/actions/get-workflow-job-id
2025-04-25T04:43:32.3229831Z with:
2025-04-25T04:43:32.3230519Z   github-token: ***
2025-04-25T04:43:32.3230944Z env:
2025-04-25T04:43:32.3231337Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:32.3232106Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:32.3233245Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:32.3234267Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:32.3236000Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:32.3237584Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:32.3238089Z   AWS_REGION: us-east-1
2025-04-25T04:43:32.3238665Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:32.3239330Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:32.3250232Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:32.3250678Z ##[endgroup]
2025-04-25T04:43:32.3279745Z ##[group]Run set -eux
2025-04-25T04:43:32.3280233Z set -eux
2025-04-25T04:43:32.3281015Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-04-25T04:43:32.3330945Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:32.3331635Z env:
2025-04-25T04:43:32.3332040Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:32.3332803Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:32.3333911Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:32.3334970Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:32.3336741Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:32.3338329Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:32.3338835Z   AWS_REGION: us-east-1
2025-04-25T04:43:32.3339411Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:32.3340091Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:32.3350669Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:32.3351336Z   GITHUB_TOKEN: ***
2025-04-25T04:43:32.3351762Z ##[endgroup]
2025-04-25T04:43:32.3437291Z + python3 .github/scripts/get_workflow_job_id.py 14656749847 gpu5966
2025-04-25T04:43:33.2588526Z setting job-id=41133945556
2025-04-25T04:43:33.2589479Z setting job-name=linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T04:43:33.2944562Z Prepare all required actions
2025-04-25T04:43:33.2945285Z Getting action download info
2025-04-25T04:43:33.4362904Z ##[group]Run ./.github/actions/filter-test-configs
2025-04-25T04:43:33.4363467Z with:
2025-04-25T04:43:33.4364138Z   github-token: ***
2025-04-25T04:43:33.4365581Z   test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu.4"}]}
2025-04-25T04:43:33.4367309Z   job-name: linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T04:43:33.4368002Z env:
2025-04-25T04:43:33.4368350Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:33.4369015Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:33.4370011Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:33.4371015Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:33.4372533Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:33.4373912Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:33.4374352Z   AWS_REGION: us-east-1
2025-04-25T04:43:33.4374811Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:33.4375383Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:33.4384717Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:33.4385111Z ##[endgroup]
2025-04-25T04:43:33.4441922Z ##[group]Run nick-fields/retry@v3.0.0
2025-04-25T04:43:33.4442419Z with:
2025-04-25T04:43:33.4442739Z   shell: bash
2025-04-25T04:43:33.4443089Z   timeout_minutes: 10
2025-04-25T04:43:33.4443460Z   max_attempts: 5
2025-04-25T04:43:33.4443834Z   retry_wait_seconds: 30
2025-04-25T04:43:33.4445057Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2025-04-25T04:43:33.4446369Z   polling_interval_seconds: 1
2025-04-25T04:43:33.4446814Z   warning_on_retry: true
2025-04-25T04:43:33.4447223Z   continue_on_error: false
2025-04-25T04:43:33.4447627Z env:
2025-04-25T04:43:33.4447963Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:33.4448616Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:33.4449582Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:33.4450501Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:33.4452011Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:33.4453385Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:33.4453831Z   AWS_REGION: us-east-1
2025-04-25T04:43:33.4454321Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:33.4454899Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:33.4464265Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:33.4464881Z   GITHUB_TOKEN: ***
2025-04-25T04:43:33.4465255Z ##[endgroup]
2025-04-25T04:43:33.5466497Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2025-04-25T04:43:33.8269310Z Defaulting to user installation because normal site-packages is not writeable
2025-04-25T04:43:33.9969961Z Requirement already satisfied: requests==2.27.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (2.27.1)
2025-04-25T04:43:33.9973844Z Requirement already satisfied: pyyaml==6.0.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (6.0.1)
2025-04-25T04:43:34.0063826Z Requirement already satisfied: charset-normalizer~=2.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12)
2025-04-25T04:43:34.0072107Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3)
2025-04-25T04:43:34.0076249Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5)
2025-04-25T04:43:34.0080918Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20)
2025-04-25T04:43:34.5468888Z Command completed after 1 attempt(s).
2025-04-25T04:43:34.5582843Z ##[group]Run set -x
2025-04-25T04:43:34.5583069Z set -x
2025-04-25T04:43:34.5583241Z 
2025-04-25T04:43:34.5583547Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-04-25T04:43:34.5583905Z # in runner workspace
2025-04-25T04:43:34.5584192Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2025-04-25T04:43:34.5609384Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:34.5609672Z env:
2025-04-25T04:43:34.5609861Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:34.5610199Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:34.5610662Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:34.5611088Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:34.5611792Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:34.5612428Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:34.5612641Z   AWS_REGION: us-east-1
2025-04-25T04:43:34.5612891Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:34.5613175Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:34.5617467Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:34.5617666Z ##[endgroup]
2025-04-25T04:43:34.5676705Z + python3 /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2025-04-25T04:43:34.6143881Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2025-04-25T04:43:34.6144643Z echo "Workflow: ${GITHUB_WORKFLOW}"
2025-04-25T04:43:34.6145263Z echo "Job name: ${JOB_NAME}"
2025-04-25T04:43:34.6145809Z 
2025-04-25T04:43:34.6146521Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-04-25T04:43:34.6147418Z # in runner workspace
2025-04-25T04:43:34.6148223Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2025-04-25T04:43:34.6149138Z   --workflow "${GITHUB_WORKFLOW}" \
2025-04-25T04:43:34.6149768Z   --job-name "${JOB_NAME}" \
2025-04-25T04:43:34.6151621Z   --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu.4"}]}" \
2025-04-25T04:43:34.6153517Z   --selected-test-configs "" \
2025-04-25T04:43:34.6154150Z   --pr-number "${PR_NUMBER}" \
2025-04-25T04:43:34.6154724Z   --tag "${TAG}" \
2025-04-25T04:43:34.6155286Z   --event-name "${EVENT_NAME}" \
2025-04-25T04:43:34.6155904Z   --schedule "${SCHEDULE}" \
2025-04-25T04:43:34.6156519Z   --branch "${HEAD_BRANCH}"
2025-04-25T04:43:34.6214123Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:34.6214867Z env:
2025-04-25T04:43:34.6215288Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:34.6216124Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:34.6217324Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:34.6218420Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:34.6220712Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:34.6222749Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:34.6223269Z   AWS_REGION: us-east-1
2025-04-25T04:43:34.6223851Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:34.6224568Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:34.6235804Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:34.6236541Z   GITHUB_TOKEN: ***
2025-04-25T04:43:34.6237526Z   JOB_NAME: linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T04:43:34.6238711Z   PR_NUMBER: 
2025-04-25T04:43:34.6239332Z   TAG: ciflow/trunk/148893
2025-04-25T04:43:34.6240631Z   EVENT_NAME: push
2025-04-25T04:43:34.6241229Z   SCHEDULE: 
2025-04-25T04:43:34.6241843Z   HEAD_BRANCH: 
2025-04-25T04:43:34.6242762Z ##[endgroup]
2025-04-25T04:43:34.6335245Z Workflow: trunk
2025-04-25T04:43:34.6336349Z Job name: linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T04:43:34.9555251Z INFO:root:Found no test-config label on the PR, so all test configs are included
2025-04-25T04:43:35.3289079Z ##[group]Run echo "Filtered matrix:"
2025-04-25T04:43:35.3290326Z echo "Filtered matrix:"
2025-04-25T04:43:35.3292742Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu.4"}]}"
2025-04-25T04:43:35.3294859Z 
2025-04-25T04:43:35.3295628Z echo
2025-04-25T04:43:35.3296398Z echo "Is the current job unstable? False"
2025-04-25T04:43:35.3297210Z 
2025-04-25T04:43:35.3297961Z echo
2025-04-25T04:43:35.3298710Z echo "Is keep-going label set? False"
2025-04-25T04:43:35.3299704Z 
2025-04-25T04:43:35.3300630Z echo
2025-04-25T04:43:35.3301524Z echo "Renabled issues? "
2025-04-25T04:43:35.3357473Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:35.3358332Z env:
2025-04-25T04:43:35.3359052Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:35.3360106Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:35.3361328Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:35.3362602Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:35.3364360Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:35.3366000Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:35.3366733Z   AWS_REGION: us-east-1
2025-04-25T04:43:35.3367536Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:35.3368631Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:35.3379037Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:35.3379756Z ##[endgroup]
2025-04-25T04:43:35.3475827Z Filtered matrix:
2025-04-25T04:43:35.3478028Z {include: [{config: default, shard: 1, num_shards: 2, runner: linux.rocm.gpu.2}, {config: default, shard: 2, num_shards: 2, runner: linux.rocm.gpu.2}, {config: distributed, shard: 1, num_shards: 1, runner: linux.rocm.gpu.4}]}
2025-04-25T04:43:35.3479727Z 
2025-04-25T04:43:35.3480314Z Is the current job unstable? False
2025-04-25T04:43:35.3481356Z 
2025-04-25T04:43:35.3481818Z Is keep-going label set? False
2025-04-25T04:43:35.3482399Z 
2025-04-25T04:43:35.3482708Z Renabled issues? 
2025-04-25T04:43:35.3570050Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:35.3571005Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-04-25T04:43:35.3625560Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T04:43:35.3626263Z env:
2025-04-25T04:43:35.3626671Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:35.3627448Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:35.3629037Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:35.3630104Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:35.3631884Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:35.3633443Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:35.3634020Z   AWS_REGION: us-east-1
2025-04-25T04:43:35.3634582Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:35.3635252Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:35.3646403Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:35.3646871Z   JOB_TIMEOUT: 300
2025-04-25T04:43:35.3647290Z ##[endgroup]
2025-04-25T04:43:35.3831379Z ##[group]Run set -x
2025-04-25T04:43:35.3832011Z set -x
2025-04-25T04:43:35.3832453Z 
2025-04-25T04:43:35.3832981Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2025-04-25T04:43:35.3833736Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2025-04-25T04:43:35.3834485Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2025-04-25T04:43:35.3835174Z   TEST_COMMAND=.ci/caffe2/test.sh
2025-04-25T04:43:35.3835750Z else
2025-04-25T04:43:35.3836229Z   TEST_COMMAND=.ci/pytorch/test.sh
2025-04-25T04:43:35.3836793Z fi
2025-04-25T04:43:35.3837186Z 
2025-04-25T04:43:35.3837810Z # detached container should get cleaned up by teardown_ec2_linux
2025-04-25T04:43:35.3838806Z # TODO: Stop building test binaries as part of the build phase
2025-04-25T04:43:35.3839808Z # Used for GPU_FLAG since that doesn't play nice
2025-04-25T04:43:35.3840600Z # shellcheck disable=SC2086,SC2090
2025-04-25T04:43:35.3841238Z container_name=$(docker run \
2025-04-25T04:43:35.3841828Z   ${GPU_FLAG:-} \
2025-04-25T04:43:35.3842373Z   -e BUILD_ENVIRONMENT \
2025-04-25T04:43:35.3842942Z   -e PR_NUMBER \
2025-04-25T04:43:35.3843453Z   -e GITHUB_ACTIONS \
2025-04-25T04:43:35.3843994Z   -e GITHUB_REPOSITORY \
2025-04-25T04:43:35.3844539Z   -e GITHUB_WORKFLOW \
2025-04-25T04:43:35.3845068Z   -e GITHUB_JOB \
2025-04-25T04:43:35.3845569Z   -e GITHUB_RUN_ID \
2025-04-25T04:43:35.3846068Z   -e GITHUB_RUN_NUMBER \
2025-04-25T04:43:35.3846612Z   -e GITHUB_RUN_ATTEMPT \
2025-04-25T04:43:35.3847148Z   -e JOB_ID \
2025-04-25T04:43:35.3847615Z   -e JOB_NAME \
2025-04-25T04:43:35.3848085Z   -e BRANCH \
2025-04-25T04:43:35.3848539Z   -e SHA1 \
2025-04-25T04:43:35.3849010Z   -e AWS_DEFAULT_REGION \
2025-04-25T04:43:35.3849569Z   -e IN_WHEEL_TEST \
2025-04-25T04:43:35.3850083Z   -e SHARD_NUMBER \
2025-04-25T04:43:35.3850600Z   -e TEST_CONFIG \
2025-04-25T04:43:35.3851115Z   -e NUM_TEST_SHARDS \
2025-04-25T04:43:35.3851649Z   -e REENABLED_ISSUES \
2025-04-25T04:43:35.3852211Z   -e CONTINUE_THROUGH_ERROR \
2025-04-25T04:43:35.3852794Z   -e VERBOSE_TEST_LOGS \
2025-04-25T04:43:35.3853328Z   -e TEST_SHOWLOCALS \
2025-04-25T04:43:35.3853848Z   -e NO_TEST_TIMEOUT \
2025-04-25T04:43:35.3854349Z   -e NO_TD \
2025-04-25T04:43:35.3854876Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2025-04-25T04:43:35.3855545Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2025-04-25T04:43:35.3856216Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2025-04-25T04:43:35.3856837Z   -e TESTS_TO_INCLUDE \
2025-04-25T04:43:35.3857380Z   -e DASHBOARD_TAG \
2025-04-25T04:43:35.3858058Z   --env-file="${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" \
2025-04-25T04:43:35.3858826Z   --ulimit stack=10485760:83886080 \
2025-04-25T04:43:35.3859421Z   --ulimit core=0 \
2025-04-25T04:43:35.3860474Z   --security-opt seccomp=unconfined \
2025-04-25T04:43:35.3861112Z   --cap-add=SYS_PTRACE \
2025-04-25T04:43:35.3861652Z   --shm-size="8g" \
2025-04-25T04:43:35.3862139Z   --tty \
2025-04-25T04:43:35.3862581Z   --detach \
2025-04-25T04:43:35.3863060Z   --name="${container_name}" \
2025-04-25T04:43:35.3863645Z   --user jenkins \
2025-04-25T04:43:35.3864293Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2025-04-25T04:43:35.3865034Z   -w /var/lib/jenkins/workspace \
2025-04-25T04:43:35.3865614Z   "${DOCKER_IMAGE}"
2025-04-25T04:43:35.3866094Z )
2025-04-25T04:43:35.3866559Z # save container name for later step
2025-04-25T04:43:35.3867763Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV"
2025-04-25T04:43:35.3869132Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home
2025-04-25T04:43:35.3870868Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}"
2025-04-25T04:43:35.3928278Z shell: /usr/bin/bash -e {0}
2025-04-25T04:43:35.3928822Z env:
2025-04-25T04:43:35.3929255Z   GIT_DEFAULT_BRANCH: main
2025-04-25T04:43:35.3930070Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T04:43:35.3931283Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T04:43:35.3932420Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T04:43:35.3934302Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T04:43:35.3935972Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T04:43:35.3936496Z   AWS_REGION: us-east-1
2025-04-25T04:43:35.3937098Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T04:43:35.3937817Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T04:43:35.3948968Z   AWS_SESSION_TOKEN: ***
2025-04-25T04:43:35.3949544Z   BUILD_ENVIRONMENT: linux-focal-rocm-py3.10
2025-04-25T04:43:35.3950157Z   PR_NUMBER: 
2025-04-25T04:43:35.3950620Z   GITHUB_REPOSITORY: pytorch/pytorch
2025-04-25T04:43:35.3951183Z   GITHUB_WORKFLOW: trunk
2025-04-25T04:43:35.3951683Z   GITHUB_JOB: test
2025-04-25T04:43:35.3952130Z   GITHUB_RUN_ID: 14656749847
2025-04-25T04:43:35.3952621Z   GITHUB_RUN_NUMBER: 124706
2025-04-25T04:43:35.3953115Z   GITHUB_RUN_ATTEMPT: 1
2025-04-25T04:43:35.3953568Z   JOB_ID: 41133945556
2025-04-25T04:43:35.3954301Z   JOB_NAME: linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T04:43:35.3955121Z   BRANCH: 
2025-04-25T04:43:35.3955599Z   SHA1: b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:43:35.3956233Z   CONTINUE_THROUGH_ERROR: False
2025-04-25T04:43:35.3956778Z   VERBOSE_TEST_LOGS: False
2025-04-25T04:43:35.3957293Z   TEST_SHOWLOCALS: False
2025-04-25T04:43:35.3957767Z   NO_TEST_TIMEOUT: False
2025-04-25T04:43:35.3958210Z   NO_TD: False
2025-04-25T04:43:35.3958643Z   TEST_CONFIG: distributed
2025-04-25T04:43:35.3959110Z   SHARD_NUMBER: 1
2025-04-25T04:43:35.3959717Z   NUM_TEST_SHARDS: 1
2025-04-25T04:43:35.3960161Z   REENABLED_ISSUES: 
2025-04-25T04:43:35.3961359Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:35.3962673Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2025-04-25T04:43:35.3963270Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2025-04-25T04:43:35.3964860Z   TESTS_TO_INCLUDE: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl
2025-04-25T04:43:35.3966403Z   DASHBOARD_TAG: 
2025-04-25T04:43:35.3966833Z ##[endgroup]
2025-04-25T04:43:35.4060354Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2025-04-25T04:43:35.4061830Z + [[ linux-focal-rocm-py3.10 == *onnx* ]]
2025-04-25T04:43:35.4062484Z + TEST_COMMAND=.ci/pytorch/test.sh
2025-04-25T04:43:35.4081719Z +++ nproc --ignore=2
2025-04-25T04:43:35.4120160Z ++ docker run --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE -e DASHBOARD_TAG --env-file=/var/home/pytorchci/actions-runner/_work/_temp/github_env_14656749847 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /var/home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-25T04:43:35.6443763Z + container_name=e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T04:43:35.6445286Z + echo CONTAINER_NAME=e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T04:43:35.6447480Z + docker exec -t e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh'
2025-04-25T04:43:49.1451018Z Processing ./dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl
2025-04-25T04:43:49.6763660Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef) (3.16.1)
2025-04-25T04:43:49.6765889Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef) (4.13.2)
2025-04-25T04:43:49.8549052Z Collecting sympy>=1.13.3 (from torch==2.8.0a0+gitb68c0ef)
2025-04-25T04:43:49.8575946Z   Using cached sympy-1.13.3-py3-none-any.whl.metadata (12 kB)
2025-04-25T04:43:49.8608259Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef) (2.8.8)
2025-04-25T04:43:49.8614055Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef) (3.1.6)
2025-04-25T04:43:49.8620184Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef) (2024.10.0)
2025-04-25T04:43:49.8638296Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.8.0a0+gitb68c0ef) (1.3.0)
2025-04-25T04:43:49.9094298Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.8.0a0+gitb68c0ef) (3.0.2)
2025-04-25T04:43:49.9153384Z Using cached sympy-1.13.3-py3-none-any.whl (6.2 MB)
2025-04-25T04:43:50.4735601Z Installing collected packages: sympy, torch
2025-04-25T04:43:50.4736313Z   Attempting uninstall: sympy
2025-04-25T04:43:50.4745193Z     Found existing installation: sympy 1.13.1
2025-04-25T04:43:50.6107475Z     Uninstalling sympy-1.13.1:
2025-04-25T04:43:51.4863908Z       Successfully uninstalled sympy-1.13.1
2025-04-25T04:44:04.2062787Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2025-04-25T04:44:04.2064605Z timm 1.0.14 requires torchvision, which is not installed.
2025-04-25T04:44:04.2066381Z Successfully installed sympy-1.13.3 torch-2.8.0a0+gitb68c0ef
2025-04-25T04:44:04.2800331Z + export TERM=vt100
2025-04-25T04:44:04.2801037Z + TERM=vt100
2025-04-25T04:44:04.2810772Z ++ dirname .ci/pytorch/test.sh
2025-04-25T04:44:04.2839646Z + source .ci/pytorch/common.sh
2025-04-25T04:44:04.2851329Z +++ dirname .ci/pytorch/common.sh
2025-04-25T04:44:04.2878368Z ++ source .ci/pytorch/common_utils.sh
2025-04-25T04:44:04.2883991Z +++ declare -f -t trap_add
2025-04-25T04:44:04.2895250Z ++ set -ex -o pipefail
2025-04-25T04:44:04.2895884Z ++ [[ linux-focal-rocm-py3.10 == *rocm* ]]
2025-04-25T04:44:04.2896600Z ++ unset HIP_PLATFORM
2025-04-25T04:44:04.2897189Z ++ export PYTORCH_TEST_WITH_ROCM=1
2025-04-25T04:44:04.2897838Z ++ PYTORCH_TEST_WITH_ROCM=1
2025-04-25T04:44:04.2898423Z ++ export HSAKMT_DEBUG_LEVEL=4
2025-04-25T04:44:04.2899577Z ++ HSAKMT_DEBUG_LEVEL=4
2025-04-25T04:44:04.2900099Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1
2025-04-25T04:44:04.2900686Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1
2025-04-25T04:44:04.2901206Z ++ BUILD_TEST_LIBTORCH=0
2025-04-25T04:44:04.2901704Z + [[ linux-focal-rocm-py3.10 != *rocm* ]]
2025-04-25T04:44:04.2902278Z + echo 'Environment variables:'
2025-04-25T04:44:04.2902775Z Environment variables:
2025-04-25T04:44:04.2903181Z + env
2025-04-25T04:44:04.2936491Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-04-25T04:44:04.2937461Z CONTINUE_THROUGH_ERROR=False
2025-04-25T04:44:04.2938117Z BUILD_ENVIRONMENT=linux-focal-rocm-py3.10
2025-04-25T04:44:04.2938850Z HOSTNAME=gpu5966.jax.cs.cpe.ice.amd.com
2025-04-25T04:44:04.2940300Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.2941659Z GITHUB_ACTION=__self
2025-04-25T04:44:04.2942138Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-04-25T04:44:04.2942738Z GITHUB_RUN_NUMBER=124706
2025-04-25T04:44:04.2943196Z TEST_CONFIG=distributed
2025-04-25T04:44:04.2943671Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-04-25T04:44:04.2944243Z AWS_DEFAULT_REGION=us-east-1
2025-04-25T04:44:04.2944787Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2025-04-25T04:44:04.2945351Z GITHUB_REF_TYPE=tag
2025-04-25T04:44:04.2946077Z ***
2025-04-25T04:44:04.2946471Z GITHUB_REPOSITORY_ID=65600975
2025-04-25T04:44:04.2946981Z GITHUB_ACTIONS=true
2025-04-25T04:44:04.2947474Z SHA1=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:44:04.2948139Z GITHUB_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:44:04.2949132Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/148893
2025-04-25T04:44:04.2950031Z UCC_HOME=/usr
2025-04-25T04:44:04.2950427Z VERBOSE_TEST_LOGS=False
2025-04-25T04:44:04.2950910Z GITHUB_REF=refs/tags/ciflow/trunk/148893
2025-04-25T04:44:04.2951445Z SHARD_NUMBER=1
2025-04-25T04:44:04.2951868Z GITHUB_REF_PROTECTED=false
2025-04-25T04:44:04.2952326Z HOME=/var/lib/jenkins
2025-04-25T04:44:04.2952811Z GITHUB_API_URL=https://api.github.com
2025-04-25T04:44:04.2953396Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-04-25T04:44:04.2953915Z LANG=C.UTF-8
2025-04-25T04:44:04.2954397Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-04-25T04:44:04.2955014Z PYTORCH_TEST_WITH_ROCM=1
2025-04-25T04:44:04.2955459Z NUM_TEST_SHARDS=1
2025-04-25T04:44:04.2955846Z UCX_HOME=/usr
2025-04-25T04:44:04.2956886Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.2958298Z JOB_NAME=linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T04:44:04.2959096Z MAGMA_HOME=/opt/rocm/magma
2025-04-25T04:44:04.2960300Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.2961492Z HSAKMT_DEBUG_LEVEL=4
2025-04-25T04:44:04.2962463Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-04-25T04:44:04.2963496Z GITHUB_EVENT_NAME=push
2025-04-25T04:44:04.2964553Z DASHBOARD_TAG=
2025-04-25T04:44:04.2964955Z GITHUB_RUN_ID=14656749847
2025-04-25T04:44:04.2966090Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.2967308Z GITHUB_ACTOR=pytorch-bot[bot]
2025-04-25T04:44:04.2967780Z PR_NUMBER=
2025-04-25T04:44:04.2968196Z GITHUB_RUN_ATTEMPT=1
2025-04-25T04:44:04.2968732Z ANACONDA_PYTHON_VERSION=3.10
2025-04-25T04:44:04.2969420Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-04-25T04:44:04.2970131Z TERM=vt100
2025-04-25T04:44:04.2970563Z INSTALLED_VISION=yes
2025-04-25T04:44:04.2971036Z BRANCH=
2025-04-25T04:44:04.2971532Z OPENSSL_ROOT_DIR=/opt/openssl
2025-04-25T04:44:04.2973640Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl
2025-04-25T04:44:04.2975653Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-04-25T04:44:04.2976697Z GITHUB_SERVER_URL=https://github.com
2025-04-25T04:44:04.2977256Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-04-25T04:44:04.2977814Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-04-25T04:44:04.2978440Z REENABLED_ISSUES=
2025-04-25T04:44:04.2978899Z SHLVL=1
2025-04-25T04:44:04.2979307Z MAX_JOBS=126
2025-04-25T04:44:04.2979754Z GITHUB_ACTOR_ID=54816060
2025-04-25T04:44:04.2980456Z GITHUB_WORKFLOW_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:44:04.2981241Z GITHUB_REF_NAME=ciflow/trunk/148893
2025-04-25T04:44:04.2981749Z ROCM_PATH=/opt/rocm
2025-04-25T04:44:04.2982149Z GITHUB_JOB=test
2025-04-25T04:44:04.2982549Z NO_TEST_TIMEOUT=False
2025-04-25T04:44:04.2983013Z GITHUB_REPOSITORY=pytorch/pytorch
2025-04-25T04:44:04.2983510Z LC_ALL=C.UTF-8
2025-04-25T04:44:04.2983919Z GITHUB_RETENTION_DAYS=90
2025-04-25T04:44:04.2984363Z OPENSSL_DIR=/opt/openssl
2025-04-25T04:44:04.2984820Z GITHUB_ACTION_REPOSITORY=
2025-04-25T04:44:04.2986518Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-25T04:44:04.2988219Z GITHUB_BASE_REF=
2025-04-25T04:44:04.2988597Z CI=true
2025-04-25T04:44:04.2988973Z HSA_FORCE_FINE_GRAIN_PCIE=1
2025-04-25T04:44:04.2989464Z GITHUB_REPOSITORY_OWNER=pytorch
2025-04-25T04:44:04.2989939Z JOB_ID=41133945556
2025-04-25T04:44:04.2990347Z INSTALLED_PROTOBUF=yes
2025-04-25T04:44:04.2990783Z GITHUB_HEAD_REF=
2025-04-25T04:44:04.2991191Z GITHUB_ACTION_REF=
2025-04-25T04:44:04.2991606Z TEST_SHOWLOCALS=False
2025-04-25T04:44:04.3017300Z GITHUB_WORKFLOW=trunk
2025-04-25T04:44:04.3017854Z DEBIAN_FRONTEND=noninteractive
2025-04-25T04:44:04.3019283Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.3020722Z NO_TD=False
2025-04-25T04:44:04.3021208Z OLDPWD=/var/lib/jenkins
2025-04-25T04:44:04.3021666Z _=/usr/bin/env
2025-04-25T04:44:04.3022240Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2025-04-25T04:44:04.3310763Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2025-04-25T04:44:04.3311989Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-04-25T04:44:04.3313061Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2025-04-25T04:44:04.3314088Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2025-04-25T04:44:04.3314862Z + BUILD_DIR=build
2025-04-25T04:44:04.3315319Z + BUILD_RENAMED_DIR=build_renamed
2025-04-25T04:44:04.3315858Z + BUILD_BIN_DIR=build/bin
2025-04-25T04:44:04.3316323Z + SHARD_NUMBER=1
2025-04-25T04:44:04.3316737Z + NUM_TEST_SHARDS=1
2025-04-25T04:44:04.3317221Z + export TORCH_SERIALIZATION_DEBUG=1
2025-04-25T04:44:04.3317792Z + TORCH_SERIALIZATION_DEBUG=1
2025-04-25T04:44:04.3318292Z + export VALGRIND=ON
2025-04-25T04:44:04.3319222Z + VALGRIND=ON
2025-04-25T04:44:04.3319855Z + [[ linux-focal-rocm-py3.10 == *clang9* ]]
2025-04-25T04:44:04.3320497Z + [[ linux-focal-rocm-py3.10 == *xpu* ]]
2025-04-25T04:44:04.3321119Z + [[ linux-focal-rocm-py3.10 == *s390x* ]]
2025-04-25T04:44:04.3321764Z + [[ 0 == \1 ]]
2025-04-25T04:44:04.3322225Z + [[ False == \1 ]]
2025-04-25T04:44:04.3322768Z + [[ linux-focal-rocm-py3.10 != *bazel* ]]
2025-04-25T04:44:04.3327009Z ++ realpath build/custom_test_artifacts
2025-04-25T04:44:04.3357485Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts
2025-04-25T04:44:04.3359704Z + [[ -n test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl ]]
2025-04-25T04:44:04.3361403Z + echo 'Setting INCLUDE_CLAUSE'
2025-04-25T04:44:04.3362403Z Setting INCLUDE_CLAUSE
2025-04-25T04:44:04.3363949Z + INCLUDE_CLAUSE='--include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl'
2025-04-25T04:44:04.3365563Z + echo 'Environment variables'
2025-04-25T04:44:04.3366080Z Environment variables
2025-04-25T04:44:04.3366484Z + env
2025-04-25T04:44:04.3388729Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-04-25T04:44:04.3389759Z CONTINUE_THROUGH_ERROR=False
2025-04-25T04:44:04.3390406Z BUILD_ENVIRONMENT=linux-focal-rocm-py3.10
2025-04-25T04:44:04.3391192Z HOSTNAME=gpu5966.jax.cs.cpe.ice.amd.com
2025-04-25T04:44:04.3392609Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.3393780Z GITHUB_ACTION=__self
2025-04-25T04:44:04.3394261Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-04-25T04:44:04.3394822Z GITHUB_RUN_NUMBER=124706
2025-04-25T04:44:04.3395292Z TEST_CONFIG=distributed
2025-04-25T04:44:04.3395759Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-04-25T04:44:04.3396347Z AWS_DEFAULT_REGION=us-east-1
2025-04-25T04:44:04.3396886Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2025-04-25T04:44:04.3397433Z GITHUB_REF_TYPE=tag
2025-04-25T04:44:04.3397959Z ***
2025-04-25T04:44:04.3398356Z GITHUB_REPOSITORY_ID=65600975
2025-04-25T04:44:04.3398873Z GITHUB_ACTIONS=true
2025-04-25T04:44:04.3399377Z SHA1=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:44:04.3400432Z GITHUB_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:44:04.3401463Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/148893
2025-04-25T04:44:04.3402386Z UCC_HOME=/usr
2025-04-25T04:44:04.3402808Z TORCH_SERIALIZATION_DEBUG=1
2025-04-25T04:44:04.3403290Z VERBOSE_TEST_LOGS=False
2025-04-25T04:44:04.3403774Z GITHUB_REF=refs/tags/ciflow/trunk/148893
2025-04-25T04:44:04.3404325Z SHARD_NUMBER=1
2025-04-25T04:44:04.3404753Z GITHUB_REF_PROTECTED=false
2025-04-25T04:44:04.3405217Z HOME=/var/lib/jenkins
2025-04-25T04:44:04.3405703Z GITHUB_API_URL=https://api.github.com
2025-04-25T04:44:04.3406309Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-04-25T04:44:04.3406823Z LANG=C.UTF-8
2025-04-25T04:44:04.3407310Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-04-25T04:44:04.3407941Z PYTORCH_TEST_WITH_ROCM=1
2025-04-25T04:44:04.3408393Z NUM_TEST_SHARDS=1
2025-04-25T04:44:04.3408792Z UCX_HOME=/usr
2025-04-25T04:44:04.3409854Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.3411284Z JOB_NAME=linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T04:44:04.3412093Z MAGMA_HOME=/opt/rocm/magma
2025-04-25T04:44:04.3413170Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.3414283Z HSAKMT_DEBUG_LEVEL=4
2025-04-25T04:44:04.3415126Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-04-25T04:44:04.3416167Z GITHUB_EVENT_NAME=push
2025-04-25T04:44:04.3417258Z DASHBOARD_TAG=
2025-04-25T04:44:04.3417662Z GITHUB_RUN_ID=14656749847
2025-04-25T04:44:04.3418844Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.3420083Z GITHUB_ACTOR=pytorch-bot[bot]
2025-04-25T04:44:04.3420571Z PR_NUMBER=
2025-04-25T04:44:04.3420950Z GITHUB_RUN_ATTEMPT=1
2025-04-25T04:44:04.3421369Z VALGRIND=ON
2025-04-25T04:44:04.3421770Z ANACONDA_PYTHON_VERSION=3.10
2025-04-25T04:44:04.3422365Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-04-25T04:44:04.3422967Z TERM=vt100
2025-04-25T04:44:04.3423334Z INSTALLED_VISION=yes
2025-04-25T04:44:04.3423733Z BRANCH=
2025-04-25T04:44:04.3424121Z OPENSSL_ROOT_DIR=/opt/openssl
2025-04-25T04:44:04.3425924Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl
2025-04-25T04:44:04.3427940Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-04-25T04:44:04.3429043Z GITHUB_SERVER_URL=https://github.com
2025-04-25T04:44:04.3429622Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-04-25T04:44:04.3430197Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-04-25T04:44:04.3430792Z REENABLED_ISSUES=
2025-04-25T04:44:04.3431184Z SHLVL=1
2025-04-25T04:44:04.3431528Z MAX_JOBS=126
2025-04-25T04:44:04.3431916Z GITHUB_ACTOR_ID=54816060
2025-04-25T04:44:04.3432526Z GITHUB_WORKFLOW_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T04:44:04.3433232Z GITHUB_REF_NAME=ciflow/trunk/148893
2025-04-25T04:44:04.3433747Z ROCM_PATH=/opt/rocm
2025-04-25T04:44:04.3434153Z GITHUB_JOB=test
2025-04-25T04:44:04.3434554Z NO_TEST_TIMEOUT=False
2025-04-25T04:44:04.3435029Z GITHUB_REPOSITORY=pytorch/pytorch
2025-04-25T04:44:04.3435531Z LC_ALL=C.UTF-8
2025-04-25T04:44:04.3435932Z GITHUB_RETENTION_DAYS=90
2025-04-25T04:44:04.3436380Z OPENSSL_DIR=/opt/openssl
2025-04-25T04:44:04.3436850Z GITHUB_ACTION_REPOSITORY=
2025-04-25T04:44:04.3438543Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-25T04:44:04.3440394Z GITHUB_BASE_REF=
2025-04-25T04:44:04.3440774Z CI=true
2025-04-25T04:44:04.3441204Z HSA_FORCE_FINE_GRAIN_PCIE=1
2025-04-25T04:44:04.3441797Z GITHUB_REPOSITORY_OWNER=pytorch
2025-04-25T04:44:04.3442371Z JOB_ID=41133945556
2025-04-25T04:44:04.3442856Z INSTALLED_PROTOBUF=yes
2025-04-25T04:44:04.3443379Z GITHUB_HEAD_REF=
2025-04-25T04:44:04.3443767Z GITHUB_ACTION_REF=
2025-04-25T04:44:04.3444172Z TEST_SHOWLOCALS=False
2025-04-25T04:44:04.3444597Z GITHUB_WORKFLOW=trunk
2025-04-25T04:44:04.3445053Z DEBIAN_FRONTEND=noninteractive
2025-04-25T04:44:04.3446226Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_9aa418fe-3246-4ec0-82e1-1177659777bb
2025-04-25T04:44:04.3447373Z NO_TD=False
2025-04-25T04:44:04.3447755Z OLDPWD=/var/lib/jenkins
2025-04-25T04:44:04.3448178Z _=/usr/bin/env
2025-04-25T04:44:04.3448590Z + echo 'Testing pytorch'
2025-04-25T04:44:04.3449030Z Testing pytorch
2025-04-25T04:44:04.3449444Z + export LANG=C.UTF-8
2025-04-25T04:44:04.3449866Z + LANG=C.UTF-8
2025-04-25T04:44:04.3450247Z + PR_NUMBER=
2025-04-25T04:44:04.3450656Z + [[ distributed == \d\e\f\a\u\l\t ]]
2025-04-25T04:44:04.3451234Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2025-04-25T04:44:04.3451835Z + [[ linux-focal-rocm-py3.10 == *rocm* ]]
2025-04-25T04:44:04.3452422Z + export HIP_VISIBLE_DEVICES=0,1,2,3
2025-04-25T04:44:04.3452968Z + HIP_VISIBLE_DEVICES=0,1,2,3
2025-04-25T04:44:04.3453469Z + [[ distributed == \s\l\o\w ]]
2025-04-25T04:44:04.3454051Z + [[ linux-focal-rocm-py3.10 == *slow-gradcheck* ]]
2025-04-25T04:44:04.3454705Z + [[ linux-focal-rocm-py3.10 == *cuda* ]]
2025-04-25T04:44:04.3455274Z + [[ linux-focal-rocm-py3.10 == *rocm* ]]
2025-04-25T04:44:04.3455938Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-04-25T04:44:04.3457159Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-04-25T04:44:04.3457739Z + [[ distributed == *crossref* ]]
2025-04-25T04:44:04.3458281Z + [[ linux-focal-rocm-py3.10 == *rocm* ]]
2025-04-25T04:44:04.3458821Z + export VALGRIND=OFF
2025-04-25T04:44:04.3459233Z + VALGRIND=OFF
2025-04-25T04:44:04.3459603Z + rocminfo
2025-04-25T04:44:04.3619224Z ROCk module version 6.10.5 is loaded
2025-04-25T04:44:04.4805393Z =====================    
2025-04-25T04:44:04.4806062Z HSA System Attributes    
2025-04-25T04:44:04.4806595Z =====================    
2025-04-25T04:44:04.4807066Z Runtime Version:         1.14
2025-04-25T04:44:04.4807664Z Runtime Ext Version:     1.6
2025-04-25T04:44:04.4808203Z System Timestamp Freq.:  1000.000000MHz
2025-04-25T04:44:04.4809688Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-04-25T04:44:04.4810655Z Machine Model:           LARGE                              
2025-04-25T04:44:04.4811414Z System Endianness:       LITTLE                             
2025-04-25T04:44:04.4812073Z Mwaitx:                  DISABLED
2025-04-25T04:44:04.4812590Z DMAbuf Support:          YES
2025-04-25T04:44:04.4812907Z 
2025-04-25T04:44:04.4813062Z ==========               
2025-04-25T04:44:04.4813528Z HSA Agents               
2025-04-25T04:44:04.4813957Z ==========               
2025-04-25T04:44:04.4814364Z *******                  
2025-04-25T04:44:04.4814783Z Agent 1                  
2025-04-25T04:44:04.4815196Z *******                  
2025-04-25T04:44:04.4815736Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-04-25T04:44:04.4816439Z   Uuid:                    CPU-XX                             
2025-04-25T04:44:04.4817137Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-04-25T04:44:04.4817882Z   Vendor Name:             CPU                                
2025-04-25T04:44:04.4818695Z   Feature:                 None specified                     
2025-04-25T04:44:04.4819516Z   Profile:                 FULL_PROFILE                       
2025-04-25T04:44:04.4820324Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.4821071Z   Max Queue Number:        0(0x0)                             
2025-04-25T04:44:04.4821768Z   Queue Min Size:          0(0x0)                             
2025-04-25T04:44:04.4822455Z   Queue Max Size:          0(0x0)                             
2025-04-25T04:44:04.4823129Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.4823756Z   Node:                    0                                  
2025-04-25T04:44:04.4824482Z   Device Type:             CPU                                
2025-04-25T04:44:04.4825103Z   Cache Info:              
2025-04-25T04:44:04.4825609Z     L1:                      32768(0x8000) KB                   
2025-04-25T04:44:04.4826227Z   Chip ID:                 0(0x0)                             
2025-04-25T04:44:04.4826906Z   ASIC Revision:           0(0x0)                             
2025-04-25T04:44:04.4827620Z   Cacheline Size:          64(0x40)                           
2025-04-25T04:44:04.4828317Z   Max Clock Freq. (MHz):   2000                               
2025-04-25T04:44:04.4829057Z   BDFID:                   0                                  
2025-04-25T04:44:04.4829724Z   Internal Node ID:        0                                  
2025-04-25T04:44:04.4830409Z   Compute Unit:            64                                 
2025-04-25T04:44:04.4831079Z   SIMDs per CU:            0                                  
2025-04-25T04:44:04.4831762Z   Shader Engines:          0                                  
2025-04-25T04:44:04.4832521Z   Shader Arrs. per Eng.:   0                                  
2025-04-25T04:44:04.4833251Z   WatchPts on Addr. Ranges:1                                  
2025-04-25T04:44:04.4833897Z   Memory Properties:       
2025-04-25T04:44:04.4834371Z   Features:                None
2025-04-25T04:44:04.4834856Z   Pool Info:               
2025-04-25T04:44:04.4835672Z     Pool 1                   
2025-04-25T04:44:04.4836272Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.4837000Z       Size:                    528196732(0x1f7ba47c) KB           
2025-04-25T04:44:04.4837979Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4838696Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4839440Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:44:04.4840394Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4841118Z       Accessible by all:       TRUE                               
2025-04-25T04:44:04.4841859Z     Pool 2                   
2025-04-25T04:44:04.4842936Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.4843660Z       Size:                    528196732(0x1f7ba47c) KB           
2025-04-25T04:44:04.4844343Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4845035Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4845765Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:44:04.4846506Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4847217Z       Accessible by all:       TRUE                               
2025-04-25T04:44:04.4847843Z     Pool 3                   
2025-04-25T04:44:04.4848416Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-04-25T04:44:04.4849074Z       Size:                    528196732(0x1f7ba47c) KB           
2025-04-25T04:44:04.4849747Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4850461Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4851196Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:44:04.4851941Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4852647Z       Accessible by all:       TRUE                               
2025-04-25T04:44:04.4853269Z     Pool 4                   
2025-04-25T04:44:04.4853835Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.4854500Z       Size:                    528196732(0x1f7ba47c) KB           
2025-04-25T04:44:04.4855168Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4855864Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4856605Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:44:04.4857348Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4858088Z       Accessible by all:       TRUE                               
2025-04-25T04:44:04.4858821Z   ISA Info:                
2025-04-25T04:44:04.4859358Z *******                  
2025-04-25T04:44:04.4859885Z Agent 2                  
2025-04-25T04:44:04.4860388Z *******                  
2025-04-25T04:44:04.4860937Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-04-25T04:44:04.4861605Z   Uuid:                    CPU-XX                             
2025-04-25T04:44:04.4862312Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-04-25T04:44:04.4863063Z   Vendor Name:             CPU                                
2025-04-25T04:44:04.4863769Z   Feature:                 None specified                     
2025-04-25T04:44:04.4864464Z   Profile:                 FULL_PROFILE                       
2025-04-25T04:44:04.4865163Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.4865879Z   Max Queue Number:        0(0x0)                             
2025-04-25T04:44:04.4866585Z   Queue Min Size:          0(0x0)                             
2025-04-25T04:44:04.4867285Z   Queue Max Size:          0(0x0)                             
2025-04-25T04:44:04.4868317Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.4868964Z   Node:                    1                                  
2025-04-25T04:44:04.4869612Z   Device Type:             CPU                                
2025-04-25T04:44:04.4870226Z   Cache Info:              
2025-04-25T04:44:04.4870738Z     L1:                      32768(0x8000) KB                   
2025-04-25T04:44:04.4871373Z   Chip ID:                 0(0x0)                             
2025-04-25T04:44:04.4872036Z   ASIC Revision:           0(0x0)                             
2025-04-25T04:44:04.4872738Z   Cacheline Size:          64(0x40)                           
2025-04-25T04:44:04.4873448Z   Max Clock Freq. (MHz):   2000                               
2025-04-25T04:44:04.4874361Z   BDFID:                   0                                  
2025-04-25T04:44:04.4875016Z   Internal Node ID:        1                                  
2025-04-25T04:44:04.4875710Z   Compute Unit:            64                                 
2025-04-25T04:44:04.4876383Z   SIMDs per CU:            0                                  
2025-04-25T04:44:04.4877073Z   Shader Engines:          0                                  
2025-04-25T04:44:04.4877781Z   Shader Arrs. per Eng.:   0                                  
2025-04-25T04:44:04.4878572Z   WatchPts on Addr. Ranges:1                                  
2025-04-25T04:44:04.4879345Z   Memory Properties:       
2025-04-25T04:44:04.4880290Z   Features:                None
2025-04-25T04:44:04.4880865Z   Pool Info:               
2025-04-25T04:44:04.4881398Z     Pool 1                   
2025-04-25T04:44:04.4881968Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.4882649Z       Size:                    528455492(0x1f7f9744) KB           
2025-04-25T04:44:04.4883324Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4884021Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4884790Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:44:04.4885529Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4886251Z       Accessible by all:       TRUE                               
2025-04-25T04:44:04.4886870Z     Pool 2                   
2025-04-25T04:44:04.4887436Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.4888125Z       Size:                    528455492(0x1f7f9744) KB           
2025-04-25T04:44:04.4888927Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4889759Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4890629Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:44:04.4891529Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4892373Z       Accessible by all:       TRUE                               
2025-04-25T04:44:04.4893119Z     Pool 3                   
2025-04-25T04:44:04.4893792Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-04-25T04:44:04.4894536Z       Size:                    528455492(0x1f7f9744) KB           
2025-04-25T04:44:04.4895191Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4895887Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4896614Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:44:04.4897344Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4898049Z       Accessible by all:       TRUE                               
2025-04-25T04:44:04.4898645Z     Pool 4                   
2025-04-25T04:44:04.4899209Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.4899917Z       Size:                    528455492(0x1f7f9744) KB           
2025-04-25T04:44:04.4900703Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4901932Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4902783Z       Alloc Recommended Granule:4KB                                
2025-04-25T04:44:04.4903510Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4904217Z       Accessible by all:       TRUE                               
2025-04-25T04:44:04.4904815Z   ISA Info:                
2025-04-25T04:44:04.4905255Z *******                  
2025-04-25T04:44:04.4905675Z Agent 3                  
2025-04-25T04:44:04.4906093Z *******                  
2025-04-25T04:44:04.4906577Z   Name:                    gfx90a                             
2025-04-25T04:44:04.4907219Z   Uuid:                    GPU-270fbd25d3852d1b               
2025-04-25T04:44:04.4908211Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.4908952Z   Vendor Name:             AMD                                
2025-04-25T04:44:04.4909638Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:44:04.4910315Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:44:04.4911001Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.4911698Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:44:04.4912382Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:44:04.4913056Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:44:04.4913724Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.4914355Z   Node:                    2                                  
2025-04-25T04:44:04.4914984Z   Device Type:             GPU                                
2025-04-25T04:44:04.4915585Z   Cache Info:              
2025-04-25T04:44:04.4916072Z     L1:                      16(0x10) KB                        
2025-04-25T04:44:04.4916655Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:44:04.4917261Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:44:04.4917898Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:44:04.4918593Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:44:04.4919280Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:44:04.4920037Z   BDFID:                   12800                              
2025-04-25T04:44:04.4920678Z   Internal Node ID:        2                                  
2025-04-25T04:44:04.4921360Z   Compute Unit:            104                                
2025-04-25T04:44:04.4922013Z   SIMDs per CU:            4                                  
2025-04-25T04:44:04.4922687Z   Shader Engines:          8                                  
2025-04-25T04:44:04.4923392Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:44:04.4924113Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:44:04.4924852Z   Coherent Host Access:    FALSE                              
2025-04-25T04:44:04.4925469Z   Memory Properties:       
2025-04-25T04:44:04.4925976Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:44:04.4926623Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:44:04.4927328Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:44:04.4928046Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.4928784Z   Workgroup Max Size per Dimension:
2025-04-25T04:44:04.4929432Z     x                        1024(0x400)                        
2025-04-25T04:44:04.4930102Z     y                        1024(0x400)                        
2025-04-25T04:44:04.4930758Z     z                        1024(0x400)                        
2025-04-25T04:44:04.4931492Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:44:04.4932303Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:44:04.4933602Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.4934330Z   Grid Max Size per Dimension:
2025-04-25T04:44:04.4934828Z     x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.4935393Z     y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.4935954Z     z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.4936618Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:44:04.4937406Z   Packet Processor uCode:: 92                                 
2025-04-25T04:44:04.4938137Z   SDMA engine uCode::      9                                  
2025-04-25T04:44:04.4938850Z   IOMMU Support::          None                               
2025-04-25T04:44:04.4945498Z   Pool Info:               
2025-04-25T04:44:04.4945997Z     Pool 1                   
2025-04-25T04:44:04.4946571Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.4947297Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.4947997Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4948707Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4949467Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.4950234Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4950959Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.4951587Z     Pool 2                   
2025-04-25T04:44:04.4952166Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.4952846Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.4953515Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4954215Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4954958Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.4955701Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4956416Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.4957040Z     Pool 3                   
2025-04-25T04:44:04.4957609Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.4958261Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.4958922Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.4959717Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.4960451Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.4961200Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.4961926Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.4962543Z     Pool 4                   
2025-04-25T04:44:04.4963083Z       Segment:                 GROUP                              
2025-04-25T04:44:04.4963717Z       Size:                    64(0x40) KB                        
2025-04-25T04:44:04.4964369Z       Allocatable:             FALSE                              
2025-04-25T04:44:04.4965063Z       Alloc Granule:           0KB                                
2025-04-25T04:44:04.4965805Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:44:04.4966546Z       Alloc Alignment:         0KB                                
2025-04-25T04:44:04.4967272Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.4967895Z   ISA Info:                
2025-04-25T04:44:04.4968367Z     ISA 1                    
2025-04-25T04:44:04.4969070Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:44:04.4969972Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:44:04.4971247Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:44:04.4972121Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.4973016Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.4973838Z       Fast f16:                TRUE                               
2025-04-25T04:44:04.4974627Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.4975296Z       Workgroup Max Size per Dimension:
2025-04-25T04:44:04.4975877Z         x                        1024(0x400)                        
2025-04-25T04:44:04.4976469Z         y                        1024(0x400)                        
2025-04-25T04:44:04.4977335Z         z                        1024(0x400)                        
2025-04-25T04:44:04.4977986Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.4978612Z       Grid Max Size per Dimension:
2025-04-25T04:44:04.4979145Z         x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.4979741Z         y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.4980430Z         z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.4981206Z       FBarrier Max Size:       32                                 
2025-04-25T04:44:04.4981925Z *******                  
2025-04-25T04:44:04.4982446Z Agent 4                  
2025-04-25T04:44:04.4982937Z *******                  
2025-04-25T04:44:04.4983428Z   Name:                    gfx90a                             
2025-04-25T04:44:04.4984071Z   Uuid:                    GPU-da59425c5973e0bc               
2025-04-25T04:44:04.4984772Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.4985488Z   Vendor Name:             AMD                                
2025-04-25T04:44:04.4986186Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:44:04.4986883Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:44:04.4987577Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.4988278Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:44:04.4988962Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:44:04.4989635Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:44:04.4990310Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.4990946Z   Node:                    3                                  
2025-04-25T04:44:04.4991588Z   Device Type:             GPU                                
2025-04-25T04:44:04.4992197Z   Cache Info:              
2025-04-25T04:44:04.4992700Z     L1:                      16(0x10) KB                        
2025-04-25T04:44:04.4993323Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:44:04.4993946Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:44:04.4994613Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:44:04.4995314Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:44:04.4996029Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:44:04.4996683Z   BDFID:                   13568                              
2025-04-25T04:44:04.4997346Z   Internal Node ID:        3                                  
2025-04-25T04:44:04.4998037Z   Compute Unit:            104                                
2025-04-25T04:44:04.4998707Z   SIMDs per CU:            4                                  
2025-04-25T04:44:04.4999406Z   Shader Engines:          8                                  
2025-04-25T04:44:04.5000427Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:44:04.5001162Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:44:04.5002251Z   Coherent Host Access:    FALSE                              
2025-04-25T04:44:04.5002902Z   Memory Properties:       
2025-04-25T04:44:04.5003423Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:44:04.5004082Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:44:04.5004806Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:44:04.5005526Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5006183Z   Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5006739Z     x                        1024(0x400)                        
2025-04-25T04:44:04.5007319Z     y                        1024(0x400)                        
2025-04-25T04:44:04.5007870Z     z                        1024(0x400)                        
2025-04-25T04:44:04.5008874Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:44:04.5009779Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:44:04.5010633Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5011377Z   Grid Max Size per Dimension:
2025-04-25T04:44:04.5011978Z     x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5012671Z     y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5013354Z     z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5014087Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:44:04.5014811Z   Packet Processor uCode:: 92                                 
2025-04-25T04:44:04.5015462Z   SDMA engine uCode::      9                                  
2025-04-25T04:44:04.5016086Z   IOMMU Support::          None                               
2025-04-25T04:44:04.5016621Z   Pool Info:               
2025-04-25T04:44:04.5017035Z     Pool 1                   
2025-04-25T04:44:04.5017553Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.5018161Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5018779Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5019405Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5020144Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5020943Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5021704Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5022358Z     Pool 2                   
2025-04-25T04:44:04.5022929Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.5023527Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5024135Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5024760Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5025532Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5026329Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5027078Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5027625Z     Pool 3                   
2025-04-25T04:44:04.5028117Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.5028706Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5029292Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5029919Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5030572Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5031236Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5031882Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5032428Z     Pool 4                   
2025-04-25T04:44:04.5033164Z       Segment:                 GROUP                              
2025-04-25T04:44:04.5033734Z       Size:                    64(0x40) KB                        
2025-04-25T04:44:04.5034320Z       Allocatable:             FALSE                              
2025-04-25T04:44:04.5034951Z       Alloc Granule:           0KB                                
2025-04-25T04:44:04.5035618Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:44:04.5036282Z       Alloc Alignment:         0KB                                
2025-04-25T04:44:04.5036934Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5037481Z   ISA Info:                
2025-04-25T04:44:04.5037873Z     ISA 1                    
2025-04-25T04:44:04.5038690Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:44:04.5039618Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:44:04.5040399Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:44:04.5041055Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5041732Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5042355Z       Fast f16:                TRUE                               
2025-04-25T04:44:04.5042971Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5043554Z       Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5044071Z         x                        1024(0x400)                        
2025-04-25T04:44:04.5044581Z         y                        1024(0x400)                        
2025-04-25T04:44:04.5045088Z         z                        1024(0x400)                        
2025-04-25T04:44:04.5045663Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5046223Z       Grid Max Size per Dimension:
2025-04-25T04:44:04.5046696Z         x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5047222Z         y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5047734Z         z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5048316Z       FBarrier Max Size:       32                                 
2025-04-25T04:44:04.5048927Z *******                  
2025-04-25T04:44:04.5049395Z Agent 5                  
2025-04-25T04:44:04.5049835Z *******                  
2025-04-25T04:44:04.5050335Z   Name:                    gfx90a                             
2025-04-25T04:44:04.5051021Z   Uuid:                    GPU-3faa9ae703e0aa3c               
2025-04-25T04:44:04.5051772Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.5052531Z   Vendor Name:             AMD                                
2025-04-25T04:44:04.5053263Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:44:04.5053975Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:44:04.5054693Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.5055329Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:44:04.5055943Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:44:04.5056546Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:44:04.5057148Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.5057705Z   Node:                    4                                  
2025-04-25T04:44:04.5058281Z   Device Type:             GPU                                
2025-04-25T04:44:04.5058825Z   Cache Info:              
2025-04-25T04:44:04.5059273Z     L1:                      16(0x10) KB                        
2025-04-25T04:44:04.5059803Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:44:04.5060450Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:44:04.5061509Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:44:04.5062419Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:44:04.5063148Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:44:04.5063739Z   BDFID:                   4352                               
2025-04-25T04:44:04.5064362Z   Internal Node ID:        4                                  
2025-04-25T04:44:04.5064992Z   Compute Unit:            104                                
2025-04-25T04:44:04.5065588Z   SIMDs per CU:            4                                  
2025-04-25T04:44:04.5066196Z   Shader Engines:          8                                  
2025-04-25T04:44:04.5066833Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:44:04.5067796Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:44:04.5068508Z   Coherent Host Access:    FALSE                              
2025-04-25T04:44:04.5069128Z   Memory Properties:       
2025-04-25T04:44:04.5069588Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:44:04.5070225Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:44:04.5070861Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:44:04.5071527Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5072120Z   Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5072616Z     x                        1024(0x400)                        
2025-04-25T04:44:04.5073204Z     y                        1024(0x400)                        
2025-04-25T04:44:04.5073715Z     z                        1024(0x400)                        
2025-04-25T04:44:04.5074346Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:44:04.5075079Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:44:04.5075810Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5076396Z   Grid Max Size per Dimension:
2025-04-25T04:44:04.5076887Z     x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5077433Z     y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5077967Z     z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5078578Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:44:04.5079267Z   Packet Processor uCode:: 92                                 
2025-04-25T04:44:04.5080422Z   SDMA engine uCode::      9                                  
2025-04-25T04:44:04.5081067Z   IOMMU Support::          None                               
2025-04-25T04:44:04.5081633Z   Pool Info:               
2025-04-25T04:44:04.5082054Z     Pool 1                   
2025-04-25T04:44:04.5082588Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.5083212Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5083831Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5084463Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5085141Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5085821Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5086471Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5087022Z     Pool 2                   
2025-04-25T04:44:04.5087540Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.5088148Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5088772Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5089519Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5090300Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5091452Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5092220Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5092875Z     Pool 3                   
2025-04-25T04:44:04.5093466Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.5094168Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5094832Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5095463Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5096125Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5096795Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5097711Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5098273Z     Pool 4                   
2025-04-25T04:44:04.5098738Z       Segment:                 GROUP                              
2025-04-25T04:44:04.5099318Z       Size:                    64(0x40) KB                        
2025-04-25T04:44:04.5099925Z       Allocatable:             FALSE                              
2025-04-25T04:44:04.5100674Z       Alloc Granule:           0KB                                
2025-04-25T04:44:04.5101461Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:44:04.5102243Z       Alloc Alignment:         0KB                                
2025-04-25T04:44:04.5103008Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5103579Z   ISA Info:                
2025-04-25T04:44:04.5103986Z     ISA 1                    
2025-04-25T04:44:04.5104509Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:44:04.5105197Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:44:04.5105853Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:44:04.5106523Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5107196Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5107810Z       Fast f16:                TRUE                               
2025-04-25T04:44:04.5108442Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5109045Z       Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5109570Z         x                        1024(0x400)                        
2025-04-25T04:44:04.5110100Z         y                        1024(0x400)                        
2025-04-25T04:44:04.5110615Z         z                        1024(0x400)                        
2025-04-25T04:44:04.5111189Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5111769Z       Grid Max Size per Dimension:
2025-04-25T04:44:04.5112244Z         x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5112762Z         y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5113276Z         z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5113861Z       FBarrier Max Size:       32                                 
2025-04-25T04:44:04.5114409Z *******                  
2025-04-25T04:44:04.5114801Z Agent 6                  
2025-04-25T04:44:04.5115182Z *******                  
2025-04-25T04:44:04.5115628Z   Name:                    gfx90a                             
2025-04-25T04:44:04.5116209Z   Uuid:                    GPU-c930cceae5e422f1               
2025-04-25T04:44:04.5116845Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.5117480Z   Vendor Name:             AMD                                
2025-04-25T04:44:04.5118093Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:44:04.5118699Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:44:04.5119658Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.5120298Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:44:04.5120933Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:44:04.5121542Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:44:04.5122148Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.5122720Z   Node:                    5                                  
2025-04-25T04:44:04.5123296Z   Device Type:             GPU                                
2025-04-25T04:44:04.5123839Z   Cache Info:              
2025-04-25T04:44:04.5124276Z     L1:                      16(0x10) KB                        
2025-04-25T04:44:04.5124793Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:44:04.5125616Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:44:04.5126215Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:44:04.5126863Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:44:04.5127500Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:44:04.5128095Z   BDFID:                   5120                               
2025-04-25T04:44:04.5128698Z   Internal Node ID:        5                                  
2025-04-25T04:44:04.5129435Z   Compute Unit:            104                                
2025-04-25T04:44:04.5130132Z   SIMDs per CU:            4                                  
2025-04-25T04:44:04.5130842Z   Shader Engines:          8                                  
2025-04-25T04:44:04.5131614Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:44:04.5132383Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:44:04.5133162Z   Coherent Host Access:    FALSE                              
2025-04-25T04:44:04.5133841Z   Memory Properties:       
2025-04-25T04:44:04.5134388Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:44:04.5135016Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:44:04.5135671Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:44:04.5136306Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5136882Z   Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5137367Z     x                        1024(0x400)                        
2025-04-25T04:44:04.5137879Z     y                        1024(0x400)                        
2025-04-25T04:44:04.5138377Z     z                        1024(0x400)                        
2025-04-25T04:44:04.5138936Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:44:04.5139570Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:44:04.5140255Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5140927Z   Grid Max Size per Dimension:
2025-04-25T04:44:04.5141471Z     x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5142084Z     y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5142685Z     z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5143354Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:44:04.5144036Z   Packet Processor uCode:: 92                                 
2025-04-25T04:44:04.5144688Z   SDMA engine uCode::      9                                  
2025-04-25T04:44:04.5145317Z   IOMMU Support::          None                               
2025-04-25T04:44:04.5145867Z   Pool Info:               
2025-04-25T04:44:04.5146275Z     Pool 1                   
2025-04-25T04:44:04.5146780Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.5147401Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5147998Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5148930Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5149594Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5150263Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5150917Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5151471Z     Pool 2                   
2025-04-25T04:44:04.5151975Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.5152576Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5153165Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5153793Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5154707Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5155382Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5156044Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5156593Z     Pool 3                   
2025-04-25T04:44:04.5157076Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.5157668Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5158252Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5158881Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5159648Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5160316Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5183257Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5184022Z     Pool 4                   
2025-04-25T04:44:04.5184558Z       Segment:                 GROUP                              
2025-04-25T04:44:04.5185189Z       Size:                    64(0x40) KB                        
2025-04-25T04:44:04.5185820Z       Allocatable:             FALSE                              
2025-04-25T04:44:04.5186452Z       Alloc Granule:           0KB                                
2025-04-25T04:44:04.5187136Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:44:04.5187817Z       Alloc Alignment:         0KB                                
2025-04-25T04:44:04.5188483Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5189041Z   ISA Info:                
2025-04-25T04:44:04.5189460Z     ISA 1                    
2025-04-25T04:44:04.5189995Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:44:04.5190693Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:44:04.5191364Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:44:04.5192017Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5192715Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5193342Z       Fast f16:                TRUE                               
2025-04-25T04:44:04.5193954Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5194552Z       Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5195088Z         x                        1024(0x400)                        
2025-04-25T04:44:04.5195613Z         y                        1024(0x400)                        
2025-04-25T04:44:04.5196124Z         z                        1024(0x400)                        
2025-04-25T04:44:04.5196691Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5197250Z       Grid Max Size per Dimension:
2025-04-25T04:44:04.5197736Z         x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5198250Z         y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5199194Z         z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5200119Z       FBarrier Max Size:       32                                 
2025-04-25T04:44:04.5200665Z *******                  
2025-04-25T04:44:04.5201077Z Agent 7                  
2025-04-25T04:44:04.5201461Z *******                  
2025-04-25T04:44:04.5201897Z   Name:                    gfx90a                             
2025-04-25T04:44:04.5202491Z   Uuid:                    GPU-4a1afa394ae4f5e3               
2025-04-25T04:44:04.5203114Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.5203731Z   Vendor Name:             AMD                                
2025-04-25T04:44:04.5204338Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:44:04.5205255Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:44:04.5205902Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.5206544Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:44:04.5207143Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:44:04.5207755Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:44:04.5208341Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.5208896Z   Node:                    6                                  
2025-04-25T04:44:04.5209478Z   Device Type:             GPU                                
2025-04-25T04:44:04.5210016Z   Cache Info:              
2025-04-25T04:44:04.5210481Z     L1:                      16(0x10) KB                        
2025-04-25T04:44:04.5211015Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:44:04.5211554Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:44:04.5212136Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:44:04.5212765Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:44:04.5213402Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:44:04.5213974Z   BDFID:                   44544                              
2025-04-25T04:44:04.5214559Z   Internal Node ID:        6                                  
2025-04-25T04:44:04.5215164Z   Compute Unit:            104                                
2025-04-25T04:44:04.5215745Z   SIMDs per CU:            4                                  
2025-04-25T04:44:04.5216350Z   Shader Engines:          8                                  
2025-04-25T04:44:04.5216978Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:44:04.5217618Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:44:04.5218275Z   Coherent Host Access:    FALSE                              
2025-04-25T04:44:04.5218854Z   Memory Properties:       
2025-04-25T04:44:04.5219303Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:44:04.5219886Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:44:04.5220525Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:44:04.5221156Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5221735Z   Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5222221Z     x                        1024(0x400)                        
2025-04-25T04:44:04.5222717Z     y                        1024(0x400)                        
2025-04-25T04:44:04.5223226Z     z                        1024(0x400)                        
2025-04-25T04:44:04.5223783Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:44:04.5224435Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:44:04.5225056Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5225613Z   Grid Max Size per Dimension:
2025-04-25T04:44:04.5226074Z     x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5226906Z     y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5227414Z     z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5228000Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:44:04.5228684Z   Packet Processor uCode:: 92                                 
2025-04-25T04:44:04.5229330Z   SDMA engine uCode::      9                                  
2025-04-25T04:44:04.5229953Z   IOMMU Support::          None                               
2025-04-25T04:44:04.5230488Z   Pool Info:               
2025-04-25T04:44:04.5230892Z     Pool 1                   
2025-04-25T04:44:04.5231400Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.5232006Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5232830Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5233455Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5234119Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5234778Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5235415Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5235970Z     Pool 2                   
2025-04-25T04:44:04.5236467Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.5237064Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5237683Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5238297Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5238935Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5239713Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5240353Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5240913Z     Pool 3                   
2025-04-25T04:44:04.5241406Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.5241987Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5242593Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5243210Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5243852Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5244497Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5245123Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5245659Z     Pool 4                   
2025-04-25T04:44:04.5246136Z       Segment:                 GROUP                              
2025-04-25T04:44:04.5246695Z       Size:                    64(0x40) KB                        
2025-04-25T04:44:04.5247280Z       Allocatable:             FALSE                              
2025-04-25T04:44:04.5247897Z       Alloc Granule:           0KB                                
2025-04-25T04:44:04.5248532Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:44:04.5249184Z       Alloc Alignment:         0KB                                
2025-04-25T04:44:04.5249817Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5250369Z   ISA Info:                
2025-04-25T04:44:04.5250765Z     ISA 1                    
2025-04-25T04:44:04.5251280Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:44:04.5251948Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:44:04.5252609Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:44:04.5253249Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5253906Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5254818Z       Fast f16:                TRUE                               
2025-04-25T04:44:04.5255427Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5256013Z       Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5256528Z         x                        1024(0x400)                        
2025-04-25T04:44:04.5257051Z         y                        1024(0x400)                        
2025-04-25T04:44:04.5257573Z         z                        1024(0x400)                        
2025-04-25T04:44:04.5258162Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5258725Z       Grid Max Size per Dimension:
2025-04-25T04:44:04.5259256Z         x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5260199Z         y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5260820Z         z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5261516Z       FBarrier Max Size:       32                                 
2025-04-25T04:44:04.5262153Z *******                  
2025-04-25T04:44:04.5262602Z Agent 8                  
2025-04-25T04:44:04.5263039Z *******                  
2025-04-25T04:44:04.5263562Z   Name:                    gfx90a                             
2025-04-25T04:44:04.5264252Z   Uuid:                    GPU-4c986e30208c5b32               
2025-04-25T04:44:04.5264979Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.5265605Z   Vendor Name:             AMD                                
2025-04-25T04:44:04.5266203Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:44:04.5266798Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:44:04.5267408Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.5268025Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:44:04.5268636Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:44:04.5269225Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:44:04.5269810Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.5270385Z   Node:                    7                                  
2025-04-25T04:44:04.5271057Z   Device Type:             GPU                                
2025-04-25T04:44:04.5271688Z   Cache Info:              
2025-04-25T04:44:04.5272204Z     L1:                      16(0x10) KB                        
2025-04-25T04:44:04.5272823Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:44:04.5273447Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:44:04.5274036Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:44:04.5274641Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:44:04.5275247Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:44:04.5275818Z   BDFID:                   45824                              
2025-04-25T04:44:04.5276397Z   Internal Node ID:        7                                  
2025-04-25T04:44:04.5276996Z   Compute Unit:            104                                
2025-04-25T04:44:04.5277576Z   SIMDs per CU:            4                                  
2025-04-25T04:44:04.5278170Z   Shader Engines:          8                                  
2025-04-25T04:44:04.5278788Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:44:04.5279409Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:44:04.5280151Z   Coherent Host Access:    FALSE                              
2025-04-25T04:44:04.5280714Z   Memory Properties:       
2025-04-25T04:44:04.5281167Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:44:04.5281731Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:44:04.5282692Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:44:04.5283313Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5283880Z   Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5284361Z     x                        1024(0x400)                        
2025-04-25T04:44:04.5284861Z     y                        1024(0x400)                        
2025-04-25T04:44:04.5285342Z     z                        1024(0x400)                        
2025-04-25T04:44:04.5285897Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:44:04.5286518Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:44:04.5287125Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5287669Z   Grid Max Size per Dimension:
2025-04-25T04:44:04.5288393Z     x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5288909Z     y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5289421Z     z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5289995Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:44:04.5290654Z   Packet Processor uCode:: 92                                 
2025-04-25T04:44:04.5291289Z   SDMA engine uCode::      9                                  
2025-04-25T04:44:04.5291899Z   IOMMU Support::          None                               
2025-04-25T04:44:04.5292429Z   Pool Info:               
2025-04-25T04:44:04.5292825Z     Pool 1                   
2025-04-25T04:44:04.5293324Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.5293924Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5294518Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5295127Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5295773Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5296438Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5297062Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5297606Z     Pool 2                   
2025-04-25T04:44:04.5298100Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.5298692Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5299266Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5299874Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5300513Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5301164Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5301788Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5302326Z     Pool 3                   
2025-04-25T04:44:04.5302814Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.5303382Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5303954Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5304564Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5305195Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5305837Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5306456Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5306990Z     Pool 4                   
2025-04-25T04:44:04.5307448Z       Segment:                 GROUP                              
2025-04-25T04:44:04.5308007Z       Size:                    64(0x40) KB                        
2025-04-25T04:44:04.5308579Z       Allocatable:             FALSE                              
2025-04-25T04:44:04.5309470Z       Alloc Granule:           0KB                                
2025-04-25T04:44:04.5310099Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:44:04.5310740Z       Alloc Alignment:         0KB                                
2025-04-25T04:44:04.5311358Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5311896Z   ISA Info:                
2025-04-25T04:44:04.5312279Z     ISA 1                    
2025-04-25T04:44:04.5312792Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:44:04.5313459Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:44:04.5314101Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:44:04.5314963Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5315632Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5316247Z       Fast f16:                TRUE                               
2025-04-25T04:44:04.5316847Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5317431Z       Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5317938Z         x                        1024(0x400)                        
2025-04-25T04:44:04.5318457Z         y                        1024(0x400)                        
2025-04-25T04:44:04.5318963Z         z                        1024(0x400)                        
2025-04-25T04:44:04.5319800Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5320377Z       Grid Max Size per Dimension:
2025-04-25T04:44:04.5320847Z         x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5321373Z         y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5321881Z         z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5322458Z       FBarrier Max Size:       32                                 
2025-04-25T04:44:04.5323003Z *******                  
2025-04-25T04:44:04.5323377Z Agent 9                  
2025-04-25T04:44:04.5323753Z *******                  
2025-04-25T04:44:04.5324185Z   Name:                    gfx90a                             
2025-04-25T04:44:04.5324750Z   Uuid:                    GPU-39d022f9dbb71856               
2025-04-25T04:44:04.5325364Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.5325996Z   Vendor Name:             AMD                                
2025-04-25T04:44:04.5326589Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:44:04.5327187Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:44:04.5327795Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.5328419Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:44:04.5329014Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:44:04.5329726Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:44:04.5330424Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.5331079Z   Node:                    8                                  
2025-04-25T04:44:04.5331748Z   Device Type:             GPU                                
2025-04-25T04:44:04.5332376Z   Cache Info:              
2025-04-25T04:44:04.5332885Z     L1:                      16(0x10) KB                        
2025-04-25T04:44:04.5333501Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:44:04.5334137Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:44:04.5334763Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:44:04.5335103Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:44:04.5335418Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:44:04.5335840Z   BDFID:                   36352                              
2025-04-25T04:44:04.5336113Z   Internal Node ID:        8                                  
2025-04-25T04:44:04.5336394Z   Compute Unit:            104                                
2025-04-25T04:44:04.5336666Z   SIMDs per CU:            4                                  
2025-04-25T04:44:04.5336946Z   Shader Engines:          8                                  
2025-04-25T04:44:04.5337254Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:44:04.5337563Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:44:04.5337874Z   Coherent Host Access:    FALSE                              
2025-04-25T04:44:04.5338146Z   Memory Properties:       
2025-04-25T04:44:04.5338369Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:44:04.5338755Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:44:04.5339056Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:44:04.5339364Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5339638Z   Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5339878Z     x                        1024(0x400)                        
2025-04-25T04:44:04.5340119Z     y                        1024(0x400)                        
2025-04-25T04:44:04.5340357Z     z                        1024(0x400)                        
2025-04-25T04:44:04.5340615Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:44:04.5340907Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:44:04.5341190Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5341445Z   Grid Max Size per Dimension:
2025-04-25T04:44:04.5341654Z     x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5341895Z     y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5342136Z     z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5342419Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:44:04.5342732Z   Packet Processor uCode:: 92                                 
2025-04-25T04:44:04.5343034Z   SDMA engine uCode::      9                                  
2025-04-25T04:44:04.5343323Z   IOMMU Support::          None                               
2025-04-25T04:44:04.5343579Z   Pool Info:               
2025-04-25T04:44:04.5343774Z     Pool 1                   
2025-04-25T04:44:04.5344014Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.5344301Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5344575Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5344873Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5345181Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5345497Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5345793Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5346049Z     Pool 2                   
2025-04-25T04:44:04.5346284Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.5346567Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5346842Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5347131Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5347437Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5347739Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5348034Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5348291Z     Pool 3                   
2025-04-25T04:44:04.5348518Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.5348911Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5349180Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5349468Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5349768Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5350083Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5350380Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5350638Z     Pool 4                   
2025-04-25T04:44:04.5350858Z       Segment:                 GROUP                              
2025-04-25T04:44:04.5351113Z       Size:                    64(0x40) KB                        
2025-04-25T04:44:04.5351490Z       Allocatable:             FALSE                              
2025-04-25T04:44:04.5351786Z       Alloc Granule:           0KB                                
2025-04-25T04:44:04.5352100Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:44:04.5352405Z       Alloc Alignment:         0KB                                
2025-04-25T04:44:04.5352702Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5352960Z   ISA Info:                
2025-04-25T04:44:04.5353152Z     ISA 1                    
2025-04-25T04:44:04.5353395Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:44:04.5353714Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:44:04.5354016Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:44:04.5354309Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5354618Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5354903Z       Fast f16:                TRUE                               
2025-04-25T04:44:04.5355189Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5355460Z       Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5355697Z         x                        1024(0x400)                        
2025-04-25T04:44:04.5355942Z         y                        1024(0x400)                        
2025-04-25T04:44:04.5356180Z         z                        1024(0x400)                        
2025-04-25T04:44:04.5356441Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5356702Z       Grid Max Size per Dimension:
2025-04-25T04:44:04.5356923Z         x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5357163Z         y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5357410Z         z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5357683Z       FBarrier Max Size:       32                                 
2025-04-25T04:44:04.5357941Z *******                  
2025-04-25T04:44:04.5358125Z Agent 10                 
2025-04-25T04:44:04.5358305Z *******                  
2025-04-25T04:44:04.5358512Z   Name:                    gfx90a                             
2025-04-25T04:44:04.5358786Z   Uuid:                    GPU-24bb52fc4f169fad               
2025-04-25T04:44:04.5359078Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.5359382Z   Vendor Name:             AMD                                
2025-04-25T04:44:04.5359740Z   Feature:                 KERNEL_DISPATCH                    
2025-04-25T04:44:04.5360016Z   Profile:                 BASE_PROFILE                       
2025-04-25T04:44:04.5360301Z   Float Round Mode:        NEAR                               
2025-04-25T04:44:04.5360676Z   Max Queue Number:        128(0x80)                          
2025-04-25T04:44:04.5360958Z   Queue Min Size:          64(0x40)                           
2025-04-25T04:44:04.5361233Z   Queue Max Size:          131072(0x20000)                    
2025-04-25T04:44:04.5361652Z   Queue Type:              MULTI                              
2025-04-25T04:44:04.5361912Z   Node:                    9                                  
2025-04-25T04:44:04.5362177Z   Device Type:             GPU                                
2025-04-25T04:44:04.5362427Z   Cache Info:              
2025-04-25T04:44:04.5362634Z     L1:                      16(0x10) KB                        
2025-04-25T04:44:04.5362875Z     L2:                      8192(0x2000) KB                    
2025-04-25T04:44:04.5363121Z   Chip ID:                 29708(0x740c)                      
2025-04-25T04:44:04.5363391Z   ASIC Revision:           1(0x1)                             
2025-04-25T04:44:04.5363674Z   Cacheline Size:          128(0x80)                          
2025-04-25T04:44:04.5364088Z   Max Clock Freq. (MHz):   1700                               
2025-04-25T04:44:04.5364357Z   BDFID:                   37632                              
2025-04-25T04:44:04.5364638Z   Internal Node ID:        9                                  
2025-04-25T04:44:04.5364918Z   Compute Unit:            104                                
2025-04-25T04:44:04.5365189Z   SIMDs per CU:            4                                  
2025-04-25T04:44:04.5365463Z   Shader Engines:          8                                  
2025-04-25T04:44:04.5365748Z   Shader Arrs. per Eng.:   1                                  
2025-04-25T04:44:04.5366043Z   WatchPts on Addr. Ranges:4                                  
2025-04-25T04:44:04.5366335Z   Coherent Host Access:    FALSE                              
2025-04-25T04:44:04.5366597Z   Memory Properties:       
2025-04-25T04:44:04.5366806Z   Features:                KERNEL_DISPATCH 
2025-04-25T04:44:04.5367066Z   Fast F16 Operation:      TRUE                               
2025-04-25T04:44:04.5367357Z   Wavefront Size:          64(0x40)                           
2025-04-25T04:44:04.5367649Z   Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5367920Z   Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5368140Z     x                        1024(0x400)                        
2025-04-25T04:44:04.5368372Z     y                        1024(0x400)                        
2025-04-25T04:44:04.5368602Z     z                        1024(0x400)                        
2025-04-25T04:44:04.5368856Z   Max Waves Per CU:        32(0x20)                           
2025-04-25T04:44:04.5369137Z   Max Work-item Per CU:    2048(0x800)                        
2025-04-25T04:44:04.5369416Z   Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5369667Z   Grid Max Size per Dimension:
2025-04-25T04:44:04.5369879Z     x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5370120Z     y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5370355Z     z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5370628Z   Max fbarriers/Workgrp:   32                                 
2025-04-25T04:44:04.5370947Z   Packet Processor uCode:: 92                                 
2025-04-25T04:44:04.5371243Z   SDMA engine uCode::      9                                  
2025-04-25T04:44:04.5371530Z   IOMMU Support::          None                               
2025-04-25T04:44:04.5371779Z   Pool Info:               
2025-04-25T04:44:04.5371968Z     Pool 1                   
2025-04-25T04:44:04.5372200Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-04-25T04:44:04.5372482Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5372757Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5373044Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5373347Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5373652Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5374077Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5374334Z     Pool 2                   
2025-04-25T04:44:04.5374568Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-04-25T04:44:04.5374870Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5375149Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5375438Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5375743Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5376067Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5376361Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5376714Z     Pool 3                   
2025-04-25T04:44:04.5376947Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-04-25T04:44:04.5377221Z       Size:                    67092480(0x3ffc000) KB             
2025-04-25T04:44:04.5377505Z       Allocatable:             TRUE                               
2025-04-25T04:44:04.5377800Z       Alloc Granule:           4KB                                
2025-04-25T04:44:04.5378109Z       Alloc Recommended Granule:2048KB                             
2025-04-25T04:44:04.5378426Z       Alloc Alignment:         4KB                                
2025-04-25T04:44:04.5378713Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5378966Z     Pool 4                   
2025-04-25T04:44:04.5379196Z       Segment:                 GROUP                              
2025-04-25T04:44:04.5379472Z       Size:                    64(0x40) KB                        
2025-04-25T04:44:04.5379762Z       Allocatable:             FALSE                              
2025-04-25T04:44:04.5380052Z       Alloc Granule:           0KB                                
2025-04-25T04:44:04.5380363Z       Alloc Recommended Granule:0KB                                
2025-04-25T04:44:04.5380680Z       Alloc Alignment:         0KB                                
2025-04-25T04:44:04.5380981Z       Accessible by all:       FALSE                              
2025-04-25T04:44:04.5381241Z   ISA Info:                
2025-04-25T04:44:04.5381439Z     ISA 1                    
2025-04-25T04:44:04.5381709Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-04-25T04:44:04.5382025Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-04-25T04:44:04.5382335Z       Profiles:                HSA_PROFILE_BASE                   
2025-04-25T04:44:04.5382633Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5382943Z       Default Rounding Mode:   NEAR                               
2025-04-25T04:44:04.5383229Z       Fast f16:                TRUE                               
2025-04-25T04:44:04.5383508Z       Workgroup Max Size:      1024(0x400)                        
2025-04-25T04:44:04.5383789Z       Workgroup Max Size per Dimension:
2025-04-25T04:44:04.5384024Z         x                        1024(0x400)                        
2025-04-25T04:44:04.5384264Z         y                        1024(0x400)                        
2025-04-25T04:44:04.5384496Z         z                        1024(0x400)                        
2025-04-25T04:44:04.5384760Z       Grid Max Size:           4294967295(0xffffffff)             
2025-04-25T04:44:04.5385019Z       Grid Max Size per Dimension:
2025-04-25T04:44:04.5385236Z         x                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5385477Z         y                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5385720Z         z                        4294967295(0xffffffff)             
2025-04-25T04:44:04.5385987Z       FBarrier Max Size:       32                                 
2025-04-25T04:44:04.5386238Z *** Done ***             
2025-04-25T04:44:04.5386417Z + rocminfo
2025-04-25T04:44:04.5386712Z + grep -E 'Name:.*\sgfx|Marketing'
2025-04-25T04:44:04.6433385Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-04-25T04:44:04.6434284Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-04-25T04:44:04.6435020Z   Name:                    gfx90a                             
2025-04-25T04:44:04.6435773Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.6436467Z   Name:                    gfx90a                             
2025-04-25T04:44:04.6437148Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.6437818Z   Name:                    gfx90a                             
2025-04-25T04:44:04.6438508Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.6439705Z   Name:                    gfx90a                             
2025-04-25T04:44:04.6440413Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.6441096Z   Name:                    gfx90a                             
2025-04-25T04:44:04.6441785Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.6442478Z   Name:                    gfx90a                             
2025-04-25T04:44:04.6443150Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.6443824Z   Name:                    gfx90a                             
2025-04-25T04:44:04.6444490Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.6445149Z   Name:                    gfx90a                             
2025-04-25T04:44:04.6445822Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-04-25T04:44:04.6625840Z + MAYBE_ROCM=rocm/
2025-04-25T04:44:04.6627699Z + [[ linux-focal-rocm-py3.10 == *xpu* ]]
2025-04-25T04:44:04.6628583Z + [[ linux-focal-rocm-py3.10 != *-bazel-* ]]
2025-04-25T04:44:04.6629238Z + pip_install --user ninja==1.10.2
2025-04-25T04:44:04.6629958Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-25T04:44:04.6630887Z + python3 -m pip install --progress-bar off --user ninja==1.10.2
2025-04-25T04:44:05.1951993Z Collecting ninja==1.10.2
2025-04-25T04:44:05.2784815Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2025-04-25T04:44:05.3116120Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2025-04-25T04:44:05.8475516Z Installing collected packages: ninja
2025-04-25T04:44:05.8541755Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2025-04-25T04:44:05.8543514Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2025-04-25T04:44:05.8561881Z Successfully installed ninja-1.10.2
2025-04-25T04:44:05.9146280Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-25T04:44:05.9150617Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-25T04:44:05.9152958Z + [[ linux-focal-rocm-py3.10 == *aarch64* ]]
2025-04-25T04:44:05.9153575Z + install_tlparse
2025-04-25T04:44:05.9154055Z + pip_install --user tlparse==0.3.30
2025-04-25T04:44:05.9154774Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-25T04:44:05.9155648Z + python3 -m pip install --progress-bar off --user tlparse==0.3.30
2025-04-25T04:44:06.4653944Z Collecting tlparse==0.3.30
2025-04-25T04:44:06.5464662Z   Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB)
2025-04-25T04:44:06.5770622Z Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)
2025-04-25T04:44:07.1943147Z Installing collected packages: tlparse
2025-04-25T04:44:07.2241309Z Successfully installed tlparse-0.3.30
2025-04-25T04:44:07.2862694Z ++ python -m site --user-base
2025-04-25T04:44:07.3263376Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-25T04:44:07.3265724Z + [[ linux-focal-rocm-py3.10 == *asan* ]]
2025-04-25T04:44:07.3266359Z + [[ linux-focal-rocm-py3.10 == *-debug* ]]
2025-04-25T04:44:07.3266973Z + [[ linux-focal-rocm-py3.10 != *-bazel-* ]]
2025-04-25T04:44:07.3267846Z + echo 'We are not in debug mode: linux-focal-rocm-py3.10. Expect the assertion to pass'
2025-04-25T04:44:07.3269504Z We are not in debug mode: linux-focal-rocm-py3.10. Expect the assertion to pass
2025-04-25T04:44:07.3276055Z + cd test
2025-04-25T04:44:07.3276817Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2025-04-25T04:44:09.0380040Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2025-04-25T04:44:09.0381207Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2025-04-25T04:44:09.0392505Z + DYNAMO_BENCHMARK_FLAGS=()
2025-04-25T04:44:09.0393224Z + [[ distributed == *pr_time_benchmarks* ]]
2025-04-25T04:44:09.0393914Z + [[ distributed == *dynamo_eager* ]]
2025-04-25T04:44:09.0394573Z + [[ distributed == *aot_eager* ]]
2025-04-25T04:44:09.0395197Z + [[ distributed == *aot_inductor* ]]
2025-04-25T04:44:09.0395838Z + [[ distributed == *max_autotune_inductor* ]]
2025-04-25T04:44:09.0396461Z + [[ distributed == *inductor* ]]
2025-04-25T04:44:09.0397057Z + [[ distributed == *dynamic* ]]
2025-04-25T04:44:09.0397602Z + [[ distributed == *cpu* ]]
2025-04-25T04:44:09.0398222Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2025-04-25T04:44:09.0463199Z + [[ linux-focal-rocm-py3.10 == *libtorch* ]]
2025-04-25T04:44:09.0463977Z + [[ linux-focal-rocm-py3.10 == *-bazel-* ]]
2025-04-25T04:44:09.0470822Z + cd test
2025-04-25T04:44:09.0471476Z + python -c 'import torch; print(torch.__config__.show())'
2025-04-25T04:44:10.4437117Z PyTorch built with:
2025-04-25T04:44:10.4437656Z   - GCC 11.4
2025-04-25T04:44:10.4438092Z   - C++ Version: 201703
2025-04-25T04:44:10.4439093Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2025-04-25T04:44:10.4440557Z   - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-04-25T04:44:10.4441359Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-04-25T04:44:10.4441971Z   - LAPACK is enabled (usually provided by MKL)
2025-04-25T04:44:10.4442565Z   - NNPACK is enabled
2025-04-25T04:44:10.4443043Z   - CPU capability usage: AVX2
2025-04-25T04:44:10.4443549Z   - HIP Runtime 6.3.42134
2025-04-25T04:44:10.4444035Z   - MIOpen 3.3.0
2025-04-25T04:44:10.4444446Z   - Magma 2.7.2
2025-04-25T04:44:10.4452023Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.8.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, 
2025-04-25T04:44:10.4460538Z 
2025-04-25T04:44:10.7441461Z + cd test
2025-04-25T04:44:10.7442179Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2025-04-25T04:44:12.0296638Z ATen/Parallel:
2025-04-25T04:44:12.0297238Z 	at::get_num_threads() : 128
2025-04-25T04:44:12.0297802Z 	at::get_num_interop_threads() : 128
2025-04-25T04:44:12.0298409Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-04-25T04:44:12.0298975Z 	omp_get_max_threads() : 128
2025-04-25T04:44:12.0299990Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2025-04-25T04:44:12.0301034Z 	mkl_get_max_threads() : 128
2025-04-25T04:44:12.0301744Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-04-25T04:44:12.0302553Z std::thread::hardware_concurrency() : 128
2025-04-25T04:44:12.0303796Z Environment variables:
2025-04-25T04:44:12.0304314Z 	OMP_NUM_THREADS : [not set]
2025-04-25T04:44:12.0304806Z 	MKL_NUM_THREADS : [not set]
2025-04-25T04:44:12.0305321Z ATen parallel backend: OpenMP
2025-04-25T04:44:12.0305658Z 
2025-04-25T04:44:12.4434434Z + [[ distributed == *numpy_2* ]]
2025-04-25T04:44:12.4435118Z + [[ linux-focal-rocm-py3.10 == *aarch64* ]]
2025-04-25T04:44:12.4435769Z + [[ distributed == *backward* ]]
2025-04-25T04:44:12.4436356Z + [[ distributed == *xla* ]]
2025-04-25T04:44:12.4436883Z + [[ distributed == *executorch* ]]
2025-04-25T04:44:12.4437455Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2025-04-25T04:44:12.4438062Z + [[ linux-focal-rocm-py3.10 == *libtorch* ]]
2025-04-25T04:44:12.4438672Z + [[ distributed == distributed ]]
2025-04-25T04:44:12.4439195Z + test_distributed
2025-04-25T04:44:12.4440045Z + echo 'Testing distributed python tests'
2025-04-25T04:44:12.4440665Z Testing distributed python tests
2025-04-25T04:44:12.4442607Z + python test/run_test.py --distributed-tests --shard 1 1 --include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl --verbose
2025-04-25T04:44:12.5612267Z /var/lib/jenkins/pytorch/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
2025-04-25T04:44:12.5613756Z   import pkg_resources
2025-04-25T04:44:16.6972031Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-04-25T04:44:16.9262655Z Ignoring disabled issues:  ['']
2025-04-25T04:44:16.9603557Z Found test times from artifacts
2025-04-25T04:44:17.0272892Z Found test times from artifacts
2025-04-25T04:44:17.0292401Z Running all tests
2025-04-25T04:44:17.0297037Z Running parallel tests on 8 processes
2025-04-25T04:44:17.0297728Z Name: tests to run (est. time: 25.7min)
2025-04-25T04:44:17.0298355Z   Serial tests (4):
2025-04-25T04:44:17.0298863Z     distributed/test_c10d_common 1/1
2025-04-25T04:44:17.0299447Z     distributed/test_c10d_nccl 1/3
2025-04-25T04:44:17.0299995Z     distributed/test_c10d_nccl 2/3
2025-04-25T04:44:17.0300517Z     distributed/test_c10d_nccl 3/3
2025-04-25T04:44:17.0301015Z   Parallel tests (0):
2025-04-25T04:44:17.0301484Z Name: excluded (est. time: 0.0min)
2025-04-25T04:44:17.0301979Z   Serial tests (0):
2025-04-25T04:44:17.0302411Z   Parallel tests (0):
2025-04-25T04:44:17.0433411Z Running distributed/test_c10d_common 1/1 ... [2025-04-25 04:44:17.042458]
2025-04-25T04:44:17.0434404Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-04-25T04:44:17.0436979Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-04-25 04:44:17.042817]
2025-04-25T04:47:49.5030075Z 
2025-04-25T04:47:49.5035056Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_414deacea1e79c4a_.log
2025-04-25T04:47:49.5052676Z Running 27 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_abort, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_canonicalize_helper, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_shutdown, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_default_process_group, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback
2025-04-25T04:47:49.5068400Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier
2025-04-25T04:47:49.5070038Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype
2025-04-25T04:47:49.5071740Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype
2025-04-25T04:47:49.5073421Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype
2025-04-25T04:47:49.5075103Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype
2025-04-25T04:47:49.5076588Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level
2025-04-25T04:47:49.5077943Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_abort
2025-04-25T04:47:49.5079680Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr
2025-04-25T04:47:49.5081322Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config
2025-04-25T04:47:49.5082969Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_canonicalize_helper
2025-04-25T04:47:49.5084996Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives
2025-04-25T04:47:49.5086610Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name
2025-04-25T04:47:49.5088417Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends
2025-04-25T04:47:49.5090275Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available
2025-04-25T04:47:49.5091887Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv
2025-04-25T04:47:49.5093961Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_shutdown
2025-04-25T04:47:49.5096033Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_default_process_group
2025-04-25T04:47:49.5098148Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends
2025-04-25T04:47:49.5099285Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2025-04-25T04:47:49.5100208Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop
2025-04-25T04:47:49.5100931Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable
2025-04-25T04:47:49.5101620Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal
2025-04-25T04:47:49.5102303Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle
2025-04-25T04:47:49.5102985Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank
2025-04-25T04:47:49.5103741Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback
2025-04-25T04:47:49.5104475Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv
2025-04-25T04:47:49.5105168Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback
2025-04-25T04:47:49.5105582Z 
2025-04-25T04:47:49.5105796Z Running distributed/test_c10d_nccl 1/3 ... [2025-04-25 04:47:49.503835]
2025-04-25T04:47:49.5106190Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-04-25T04:47:49.5107173Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-04-25 04:47:49.504469]
2025-04-25T05:03:18.1766404Z 
2025-04-25T05:03:18.1772673Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_be7109d15b2693b2_.log
2025-04-25T05:03:18.1830698Z Running 84 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_error_detection_and_propagation, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_restart_pg_after_error, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale
2025-04-25T05:03:18.1889162Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors
2025-04-25T05:03:18.1890717Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus
2025-04-25T05:03:18.1892699Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs
2025-04-25T05:03:18.1894585Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg
2025-04-25T05:03:18.1896207Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered
2025-04-25T05:03:18.1898202Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False
2025-04-25T05:03:18.1903839Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True
2025-04-25T05:03:18.1905483Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group
2025-04-25T05:03:18.1907207Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl
2025-04-25T05:03:18.1908869Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check
2025-04-25T05:03:18.1910894Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout
2025-04-25T05:03:18.1912577Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16
2025-04-25T05:03:18.1914144Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2
2025-04-25T05:03:18.1915751Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True
2025-04-25T05:03:18.1917321Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init
2025-04-25T05:03:18.1918910Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module
2025-04-25T05:03:18.1920775Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view
2025-04-25T05:03:18.1922391Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output
2025-04-25T05:03:18.1924319Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing
2025-04-25T05:03:18.1926362Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False
2025-04-25T05:03:18.1928313Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False
2025-04-25T05:03:18.1930295Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl
2025-04-25T05:03:18.1932345Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params
2025-04-25T05:03:18.1934332Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters
2025-04-25T05:03:18.1936174Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view
2025-04-25T05:03:18.1938021Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info
2025-04-25T05:03:18.1939867Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off
2025-04-25T05:03:18.1942078Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail
2025-04-25T05:03:18.1944303Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off
2025-04-25T05:03:18.1945999Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16
2025-04-25T05:03:18.1947538Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl
2025-04-25T05:03:18.1949492Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view
2025-04-25T05:03:18.1951227Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess
2025-04-25T05:03:18.1953046Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module
2025-04-25T05:03:18.1954688Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module
2025-04-25T05:03:18.1956883Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None
2025-04-25T05:03:18.1958892Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids
2025-04-25T05:03:18.1961117Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason
2025-04-25T05:03:18.1962968Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad
2025-04-25T05:03:18.1964554Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error
2025-04-25T05:03:18.1966232Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl
2025-04-25T05:03:18.1967665Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object
2025-04-25T05:03:18.1969001Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast
2025-04-25T05:03:18.1970259Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq
2025-04-25T05:03:18.1971598Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp
2025-04-25T05:03:18.1973209Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_error_detection_and_propagation
2025-04-25T05:03:18.1974927Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env
2025-04-25T05:03:18.1976519Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit
2025-04-25T05:03:18.1977945Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm
2025-04-25T05:03:18.1979401Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier
2025-04-25T05:03:18.1980805Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_restart_pg_after_error
2025-04-25T05:03:18.1982327Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors
2025-04-25T05:03:18.1983841Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config
2025-04-25T05:03:18.1985106Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced
2025-04-25T05:03:18.1986398Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default
2025-04-25T05:03:18.1987725Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup
2025-04-25T05:03:18.1989020Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group
2025-04-25T05:03:18.1990194Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor
2025-04-25T05:03:18.1991890Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base
2025-04-25T05:03:18.1993630Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2
2025-04-25T05:03:18.1995517Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2025-04-25T05:03:18.1997216Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True
2025-04-25T05:03:18.1999044Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False
2025-04-25T05:03:18.2000823Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True
2025-04-25T05:03:18.2002324Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True
2025-04-25T05:03:18.2003738Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False
2025-04-25T05:03:18.2005404Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False
2025-04-25T05:03:18.2006818Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True
2025-04-25T05:03:18.2008302Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False
2025-04-25T05:03:18.2009955Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False
2025-04-25T05:03:18.2011771Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False
2025-04-25T05:03:18.2013597Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False
2025-04-25T05:03:18.2015267Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata
2025-04-25T05:03:18.2016648Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_False
2025-04-25T05:03:18.2017921Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe
2025-04-25T05:03:18.2019259Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True
2025-04-25T05:03:18.2020572Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long
2025-04-25T05:03:18.2022174Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False
2025-04-25T05:03:18.2024058Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False
2025-04-25T05:03:18.2025734Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True
2025-04-25T05:03:18.2027355Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False
2025-04-25T05:03:18.2028983Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True
2025-04-25T05:03:18.2030450Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired
2025-04-25T05:03:18.2032216Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale
2025-04-25T05:03:18.2033104Z 
2025-04-25T05:03:18.2033447Z Running distributed/test_c10d_nccl 2/3 ... [2025-04-25 05:03:18.178909]
2025-04-25T05:03:18.2034133Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-04-25T05:03:18.2035824Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-04-25 05:03:18.179455]
2025-04-25T05:15:09.9109829Z 
2025-04-25T05:15:09.9115318Z distributed/test_c10d_nccl 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.3_0a50b32fb2e705f5_.log
2025-04-25T05:15:09.9169655Z Running 77 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_scalable_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_cuda_event_cache_mthd_race, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_unwaited, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_default_process_group, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group
2025-04-25T05:15:09.9223595Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl
2025-04-25T05:15:09.9225068Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_scalable_init
2025-04-25T05:15:09.9226595Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg
2025-04-25T05:15:09.9228724Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup
2025-04-25T05:15:09.9230330Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_cuda_event_cache_mthd_race
2025-04-25T05:15:09.9232006Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg
2025-04-25T05:15:09.9233706Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0
2025-04-25T05:15:09.9235318Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16
2025-04-25T05:15:09.9236823Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64
2025-04-25T05:15:09.9238399Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn
2025-04-25T05:15:09.9240091Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check
2025-04-25T05:15:09.9241707Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter
2025-04-25T05:15:09.9243522Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False
2025-04-25T05:15:09.9245114Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p
2025-04-25T05:15:09.9246714Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0
2025-04-25T05:15:09.9248366Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc
2025-04-25T05:15:09.9250296Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True
2025-04-25T05:15:09.9252446Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view
2025-04-25T05:15:09.9254372Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value
2025-04-25T05:15:09.9256072Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl
2025-04-25T05:15:09.9257856Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view
2025-04-25T05:15:09.9259689Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module
2025-04-25T05:15:09.9261532Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False
2025-04-25T05:15:09.9263632Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False
2025-04-25T05:15:09.9266294Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view
2025-04-25T05:15:09.9268218Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph
2025-04-25T05:15:09.9270103Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl
2025-04-25T05:15:09.9271892Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config
2025-04-25T05:15:09.9273848Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing
2025-04-25T05:15:09.9275520Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl
2025-04-25T05:15:09.9277313Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail
2025-04-25T05:15:09.9279121Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule
2025-04-25T05:15:09.9280904Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state
2025-04-25T05:15:09.9282947Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward
2025-04-25T05:15:09.9284861Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list
2025-04-25T05:15:09.9286874Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list
2025-04-25T05:15:09.9289035Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None
2025-04-25T05:15:09.9291104Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg
2025-04-25T05:15:09.9293066Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view
2025-04-25T05:15:09.9294725Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops
2025-04-25T05:15:09.9296278Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit
2025-04-25T05:15:09.9297906Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill
2025-04-25T05:15:09.9299697Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking
2025-04-25T05:15:09.9301390Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl
2025-04-25T05:15:09.9302808Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl
2025-04-25T05:15:09.9304170Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl
2025-04-25T05:15:09.9305456Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier
2025-04-25T05:15:09.9306746Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids
2025-04-25T05:15:09.9308130Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off
2025-04-25T05:15:09.9309497Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership
2025-04-25T05:15:09.9311328Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream
2025-04-25T05:15:09.9312746Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k
2025-04-25T05:15:09.9314137Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors
2025-04-25T05:15:09.9315439Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_unwaited
2025-04-25T05:15:09.9316937Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives
2025-04-25T05:15:09.9319139Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_default_process_group
2025-04-25T05:15:09.9321328Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False
2025-04-25T05:15:09.9323095Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True
2025-04-25T05:15:09.9324823Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False
2025-04-25T05:15:09.9326374Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True
2025-04-25T05:15:09.9328176Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False
2025-04-25T05:15:09.9330033Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False
2025-04-25T05:15:09.9331786Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync
2025-04-25T05:15:09.9333428Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True
2025-04-25T05:15:09.9335011Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True
2025-04-25T05:15:09.9336633Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True
2025-04-25T05:15:09.9338203Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True
2025-04-25T05:15:09.9340087Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False
2025-04-25T05:15:09.9342002Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False
2025-04-25T05:15:09.9343674Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True
2025-04-25T05:15:09.9345297Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False
2025-04-25T05:15:09.9346865Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True
2025-04-25T05:15:09.9348443Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True
2025-04-25T05:15:09.9350039Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False
2025-04-25T05:15:09.9351648Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True
2025-04-25T05:15:09.9353270Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False
2025-04-25T05:15:09.9355169Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group
2025-04-25T05:15:09.9356041Z 
2025-04-25T05:15:09.9356383Z Running distributed/test_c10d_nccl 3/3 ... [2025-04-25 05:15:09.913018]
2025-04-25T05:15:09.9357074Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-04-25T05:15:09.9358771Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-04-25 05:15:09.913578]
2025-04-25T05:25:28.6426150Z 
2025-04-25T05:25:28.6428837Z distributed/test_c10d_nccl 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.3_a11d7478adb3d6a5_.log
2025-04-25T05:25:28.6477116Z Running 66 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_user_buffer_registration, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::CommTest::test_time_estimate_nccl, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True
2025-04-25T05:25:28.6524037Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str
2025-04-25T05:25:28.6525779Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs
2025-04-25T05:25:28.6527503Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup
2025-04-25T05:25:28.6529126Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context
2025-04-25T05:25:28.6530970Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid
2025-04-25T05:25:28.6532436Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32
2025-04-25T05:25:28.6533988Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error
2025-04-25T05:25:28.6535896Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init
2025-04-25T05:25:28.6537637Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg
2025-04-25T05:25:28.6539552Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl
2025-04-25T05:25:28.6541286Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False
2025-04-25T05:25:28.6543115Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view
2025-04-25T05:25:28.6544926Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl
2025-04-25T05:25:28.6546562Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig
2025-04-25T05:25:28.6548195Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param
2025-04-25T05:25:28.6550019Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True
2025-04-25T05:25:28.6552015Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False
2025-04-25T05:25:28.6554107Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True
2025-04-25T05:25:28.6556097Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True
2025-04-25T05:25:28.6557980Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing
2025-04-25T05:25:28.6560010Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True
2025-04-25T05:25:28.6562034Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True
2025-04-25T05:25:28.6563926Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl
2025-04-25T05:25:28.6565867Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence
2025-04-25T05:25:28.6567685Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery
2025-04-25T05:25:28.6569651Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info
2025-04-25T05:25:28.6571518Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view
2025-04-25T05:25:28.6573375Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view
2025-04-25T05:25:28.6575505Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed
2025-04-25T05:25:28.6577879Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input
2025-04-25T05:25:28.6579607Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input
2025-04-25T05:25:28.6581305Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier
2025-04-25T05:25:28.6582887Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort
2025-04-25T05:25:28.6584359Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout
2025-04-25T05:25:28.6586289Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_user_buffer_registration
2025-04-25T05:25:28.6587908Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce
2025-04-25T05:25:28.6589363Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument
2025-04-25T05:25:28.6590885Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail
2025-04-25T05:25:28.6592337Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info
2025-04-25T05:25:28.6593876Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors
2025-04-25T05:25:28.6595407Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl
2025-04-25T05:25:28.6596802Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex
2025-04-25T05:25:28.6598131Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch
2025-04-25T05:25:28.6599455Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_time_estimate_nccl
2025-04-25T05:25:28.6601354Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single
2025-04-25T05:25:28.6603393Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn
2025-04-25T05:25:28.6605672Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced
2025-04-25T05:25:28.6608142Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends
2025-04-25T05:25:28.6610372Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False
2025-04-25T05:25:28.6612146Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True
2025-04-25T05:25:28.6613715Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg
2025-04-25T05:25:28.6615416Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check
2025-04-25T05:25:28.6617190Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True
2025-04-25T05:25:28.6618695Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False
2025-04-25T05:25:28.6620329Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True
2025-04-25T05:25:28.6622413Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_True
2025-04-25T05:25:28.6623875Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling
2025-04-25T05:25:28.6625480Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True
2025-04-25T05:25:28.6627323Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False
2025-04-25T05:25:28.6629067Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True
2025-04-25T05:25:28.6631161Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False
2025-04-25T05:25:28.6632995Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False
2025-04-25T05:25:28.6634826Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True
2025-04-25T05:25:28.6636644Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True
2025-04-25T05:25:28.6638358Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False
2025-04-25T05:25:28.6640044Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True
2025-04-25T05:25:28.6640975Z 
2025-04-25T05:25:29.3506880Z Running test batch 'tests to run' cost 2472.32 seconds
2025-04-25T05:25:30.1731598Z 
2025-04-25T05:25:30.1732086Z real	41m17.730s
2025-04-25T05:25:30.1732558Z user	67m11.965s
2025-04-25T05:25:30.1738148Z sys	98m39.759s
2025-04-25T05:25:30.1738738Z + assert_git_not_dirty
2025-04-25T05:25:30.1739306Z + [[ linux-focal-rocm-py3.10 != *rocm* ]]
2025-04-25T05:25:30.1739951Z + [[ linux-focal-rocm-py3.10 == *cuda* ]]
2025-04-25T05:25:30.1740578Z + [[ linux-focal-rocm-py3.10 == *rocm* ]]
2025-04-25T05:25:30.1741094Z + [[ 1 == 1 ]]
2025-04-25T05:25:30.1741533Z + echo 'Testing distributed C++ tests'
2025-04-25T05:25:30.1742110Z Testing distributed C++ tests
2025-04-25T05:25:30.1746643Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-04-25T05:25:30.1780963Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-04-25T05:25:30.1816037Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-04-25T05:25:30.1817232Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-04-25T05:25:30.1818219Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest
2025-04-25T05:25:30.3000242Z /var/lib/jenkins/pytorch/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
2025-04-25T05:25:30.3001755Z   import pkg_resources
2025-04-25T05:25:34.3976017Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-04-25T05:25:34.4120428Z Found test times from artifacts
2025-04-25T05:25:34.4721025Z Found test times from artifacts
2025-04-25T05:25:34.4739576Z Running all tests
2025-04-25T05:25:34.4744139Z Running parallel tests on 8 processes
2025-04-25T05:25:34.4744827Z Name: tests to run (est. time: 0.0min)
2025-04-25T05:25:34.4745422Z   Serial tests (1):
2025-04-25T05:25:34.4745916Z     cpp/FileStoreTest 1/1
2025-04-25T05:25:34.4746424Z   Parallel tests (0):
2025-04-25T05:25:34.4746890Z Name: excluded (est. time: 0.0min)
2025-04-25T05:25:34.4747394Z   Serial tests (0):
2025-04-25T05:25:34.4747810Z   Parallel tests (0):
2025-04-25T05:25:34.4878191Z Running cpp/FileStoreTest 1/1 ... [2025-04-25 05:25:34.487061]
2025-04-25T05:25:34.4879005Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-04-25T05:25:34.4883838Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-6d62cda5eac9334b.xml', '-x', '--reruns=2'] ... [2025-04-25 05:25:34.487762]
2025-04-25T05:25:37.3620373Z 
2025-04-25T05:25:37.3622430Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_2fe8dc074584ab1a_.log
2025-04-25T05:25:37.3623687Z 
2025-04-25T05:25:39.4708022Z Running test batch 'tests to run' cost 5.0 seconds
2025-04-25T05:25:40.1099118Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest
2025-04-25T05:25:40.2245757Z /var/lib/jenkins/pytorch/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
2025-04-25T05:25:40.2247276Z   import pkg_resources
2025-04-25T05:25:44.3229218Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-04-25T05:25:44.3368992Z Found test times from artifacts
2025-04-25T05:25:44.3972441Z Found test times from artifacts
2025-04-25T05:25:44.3992891Z Running all tests
2025-04-25T05:25:44.3997821Z Running parallel tests on 8 processes
2025-04-25T05:25:44.3998503Z Name: tests to run (est. time: 0.0min)
2025-04-25T05:25:44.3999089Z   Serial tests (1):
2025-04-25T05:25:44.3999891Z     cpp/HashStoreTest 1/1
2025-04-25T05:25:44.4000424Z   Parallel tests (0):
2025-04-25T05:25:44.4000909Z Name: excluded (est. time: 0.0min)
2025-04-25T05:25:44.4001411Z   Serial tests (0):
2025-04-25T05:25:44.4001833Z   Parallel tests (0):
2025-04-25T05:25:44.4126855Z Running cpp/HashStoreTest 1/1 ... [2025-04-25 05:25:44.411651]
2025-04-25T05:25:44.4127664Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-04-25T05:25:44.4130085Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-32ef0d684d2fa2c1.xml', '-x', '--reruns=2'] ... [2025-04-25 05:25:44.412153]
2025-04-25T05:25:47.0369229Z 
2025-04-25T05:25:47.0370999Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_92ca5c56f9ef4a23_.log
2025-04-25T05:25:47.0372222Z 
2025-04-25T05:25:49.0679346Z Running test batch 'tests to run' cost 4.67 seconds
2025-04-25T05:25:49.6964020Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest
2025-04-25T05:25:49.8117477Z /var/lib/jenkins/pytorch/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
2025-04-25T05:25:49.8119089Z   import pkg_resources
2025-04-25T05:25:53.9440321Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-04-25T05:25:53.9581675Z Found test times from artifacts
2025-04-25T05:25:54.0187565Z Found test times from artifacts
2025-04-25T05:25:54.0207428Z Running all tests
2025-04-25T05:25:54.0209763Z Running parallel tests on 8 processes
2025-04-25T05:25:54.0211370Z Name: tests to run (est. time: 0.0min)
2025-04-25T05:25:54.0211974Z   Serial tests (1):
2025-04-25T05:25:54.0212441Z     cpp/TCPStoreTest 1/1
2025-04-25T05:25:54.0212928Z   Parallel tests (0):
2025-04-25T05:25:54.0213393Z Name: excluded (est. time: 0.0min)
2025-04-25T05:25:54.0213900Z   Serial tests (0):
2025-04-25T05:25:54.0214304Z   Parallel tests (0):
2025-04-25T05:25:54.0343823Z Running cpp/TCPStoreTest 1/1 ... [2025-04-25 05:25:54.033584]
2025-04-25T05:25:54.0344683Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-04-25T05:25:54.0347160Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-ccf4aec959f39e06.xml', '-x', '--reruns=2'] ... [2025-04-25 05:25:54.034126]
2025-04-25T05:26:06.5829459Z 
2025-04-25T05:26:06.5831891Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_382f5e158c9796bb_.log
2025-04-25T05:26:06.5833230Z 
2025-04-25T05:26:07.4187149Z Running test batch 'tests to run' cost 13.4 seconds
2025-04-25T05:26:08.0663271Z + echo 'Testing multi-GPU linalg tests'
2025-04-25T05:26:08.0664150Z Testing multi-GPU linalg tests
2025-04-25T05:26:08.0665430Z + python test/run_test.py -i test_linalg.py -k test_matmul_offline_mgpu_tunable --verbose
2025-04-25T05:26:08.1809110Z /var/lib/jenkins/pytorch/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
2025-04-25T05:26:08.1811309Z   import pkg_resources
2025-04-25T05:26:12.3947333Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-04-25T05:26:12.4091402Z Found test times from artifacts
2025-04-25T05:26:12.4696841Z Found test times from artifacts
2025-04-25T05:26:12.4716962Z Running all tests
2025-04-25T05:26:12.4718773Z Running parallel tests on 8 processes
2025-04-25T05:26:12.4719973Z Name: tests to run (est. time: 0.21min)
2025-04-25T05:26:12.4720595Z   Serial tests (1):
2025-04-25T05:26:12.4721046Z     test_linalg 1/1
2025-04-25T05:26:12.4721502Z   Parallel tests (0):
2025-04-25T05:26:12.4721983Z Name: excluded (est. time: 0.0min)
2025-04-25T05:26:12.4722506Z   Serial tests (0):
2025-04-25T05:26:12.4722922Z   Parallel tests (0):
2025-04-25T05:26:12.4851177Z Running test_linalg 1/1 ... [2025-04-25 05:26:12.484427]
2025-04-25T05:26:12.4851941Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-04-25T05:26:12.4857416Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_linalg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_matmul_offline_mgpu_tunable', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-04-25 05:26:12.484803]
2025-04-25T05:26:28.9416183Z 
2025-04-25T05:26:28.9417516Z test_linalg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_linalg_1.1_cb4b8777faf4c07c_.log
2025-04-25T05:26:28.9419335Z Running 1 items in this shard: test/test_linalg.py::TestLinalgCUDA::test_matmul_offline_mgpu_tunableop_cuda_float32
2025-04-25T05:26:28.9420200Z 
2025-04-25T05:26:29.8053971Z Running test batch 'tests to run' cost 17.33 seconds
2025-04-25T05:26:30.6449772Z + [[ linux-focal-rocm-py3.10 == *cuda* ]]
2025-04-25T05:26:30.6450729Z + [[ 1 == 1 ]]
2025-04-25T05:26:30.6451248Z + test_rpc
2025-04-25T05:26:30.6451774Z + echo 'Testing RPC C++ tests'
2025-04-25T05:26:30.6452406Z Testing RPC C++ tests
2025-04-25T05:26:30.6458242Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-04-25T05:26:30.6500142Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-04-25T05:26:30.6538602Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-04-25T05:26:30.6540168Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc
2025-04-25T05:26:30.7710774Z /var/lib/jenkins/pytorch/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
2025-04-25T05:26:30.7712759Z   import pkg_resources
2025-04-25T05:26:34.8701372Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-04-25T05:26:34.8839850Z Found test times from artifacts
2025-04-25T05:26:34.9438509Z Found test times from artifacts
2025-04-25T05:26:34.9458646Z Running all tests
2025-04-25T05:26:34.9461068Z Running parallel tests on 8 processes
2025-04-25T05:26:34.9461734Z Name: tests to run (est. time: 0.0min)
2025-04-25T05:26:34.9462349Z   Serial tests (1):
2025-04-25T05:26:34.9462818Z     cpp/test_cpp_rpc 1/1
2025-04-25T05:26:34.9463302Z   Parallel tests (0):
2025-04-25T05:26:34.9463795Z Name: excluded (est. time: 0.0min)
2025-04-25T05:26:34.9464329Z   Serial tests (0):
2025-04-25T05:26:34.9464772Z   Parallel tests (0):
2025-04-25T05:26:34.9589247Z Running cpp/test_cpp_rpc 1/1 ... [2025-04-25 05:26:34.958061]
2025-04-25T05:26:34.9590144Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-04-25T05:26:34.9592605Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-5d200d797a56abda.xml', '-x', '--reruns=2'] ... [2025-04-25 05:26:34.958585]
2025-04-25T05:26:36.3780154Z 
2025-04-25T05:26:36.3782222Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_6e4619a32b4e102a_.log
2025-04-25T05:26:36.3783399Z 
2025-04-25T05:26:39.5443084Z Running test batch 'tests to run' cost 4.6 seconds
2025-04-25T05:26:40.1596235Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-04-25T05:26:40.1597748Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-04-25T05:26:40.1599658Z docker exec -t "e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test"
2025-04-25T05:26:40.1657760Z shell: /usr/bin/bash -e {0}
2025-04-25T05:26:40.1658318Z env:
2025-04-25T05:26:40.1658798Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:40.1659708Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:40.1661058Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:40.1662147Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:40.1663972Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:40.1665523Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:40.1666030Z   AWS_REGION: us-east-1
2025-04-25T05:26:40.1666614Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:40.1667278Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:40.1677946Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:40.1678720Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:40.1679960Z ##[endgroup]
2025-04-25T05:26:40.2950749Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2025-04-25T05:26:40.3467295Z ##[group]Run cat test/**/*_toprint.log || true
2025-04-25T05:26:40.3468041Z cat test/**/*_toprint.log || true
2025-04-25T05:26:40.3526742Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:40.3527424Z env:
2025-04-25T05:26:40.3527829Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:40.3528607Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:40.3529744Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:40.3530804Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:40.3532554Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:40.3534102Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:40.3534607Z   AWS_REGION: us-east-1
2025-04-25T05:26:40.3535275Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:40.3535957Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:40.3546582Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:40.3547354Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:40.3548186Z ##[endgroup]
2025-04-25T05:26:40.3808502Z cat: 'test/**/*_toprint.log': No such file or directory
2025-04-25T05:26:40.4012606Z Prepare all required actions
2025-04-25T05:26:40.4013601Z Getting action download info
2025-04-25T05:26:40.6278414Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2025-04-25T05:26:41.2581342Z ##[group]Run ./.github/actions/upload-test-artifacts
2025-04-25T05:26:41.2581642Z with:
2025-04-25T05:26:41.2581807Z   use-gha: true
2025-04-25T05:26:41.2582072Z   file-suffix: test-distributed-1-1-linux.rocm.gpu.4_41133945556
2025-04-25T05:26:41.2582396Z   s3-bucket: gha-artifacts
2025-04-25T05:26:41.2582602Z env:
2025-04-25T05:26:41.2582795Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:41.2583105Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:41.2583565Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:41.2584002Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:41.2584780Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:41.2585431Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:41.2585653Z   AWS_REGION: us-east-1
2025-04-25T05:26:41.2585911Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:41.2586205Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:41.2590528Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:41.2590861Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:41.2591209Z ##[endgroup]
2025-04-25T05:26:41.2659535Z ##[group]Run actions/upload-artifact@v4
2025-04-25T05:26:41.2659804Z with:
2025-04-25T05:26:41.2660130Z   name: test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_41133945556.zip
2025-04-25T05:26:41.2660495Z   retention-days: 14
2025-04-25T05:26:41.2660683Z   if-no-files-found: warn
2025-04-25T05:26:41.2660875Z   path: test/**/*.json
2025-04-25T05:26:41.2661061Z   compression-level: 6
2025-04-25T05:26:41.2661238Z   overwrite: false
2025-04-25T05:26:41.2661418Z   include-hidden-files: false
2025-04-25T05:26:41.2661609Z env:
2025-04-25T05:26:41.2661765Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:41.2662063Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:41.2662506Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:41.2662920Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:41.2663617Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:41.2664474Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:41.2664678Z   AWS_REGION: us-east-1
2025-04-25T05:26:41.2664900Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:41.2665171Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:41.2669448Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:41.2669768Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:41.2670105Z ##[endgroup]
2025-04-25T05:26:41.9800930Z With the provided path, there will be 11 files uploaded
2025-04-25T05:26:41.9807288Z Artifact name is valid!
2025-04-25T05:26:41.9807860Z Root directory input is valid!
2025-04-25T05:26:42.1440582Z Beginning upload of artifact content to blob storage
2025-04-25T05:26:42.3762276Z Uploaded bytes 44029
2025-04-25T05:26:42.4172049Z Finished uploading artifact content to blob storage!
2025-04-25T05:26:42.4176917Z SHA256 digest of uploaded artifact zip is 23aa897a677d2bdf9bf16f1d28328813b86b293dc59e8fca26cd591ddf93f7e0
2025-04-25T05:26:42.4178867Z Finalizing artifact upload
2025-04-25T05:26:42.5173857Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_41133945556.zip.zip successfully finalized. Artifact ID 3007208994
2025-04-25T05:26:42.5177117Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_41133945556.zip has been successfully uploaded! Final size is 44029 bytes. Artifact ID is 3007208994
2025-04-25T05:26:42.5186579Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/14656749847/artifacts/3007208994
2025-04-25T05:26:42.5558365Z ##[group]Run actions/upload-artifact@v4
2025-04-25T05:26:42.5558927Z with:
2025-04-25T05:26:42.5560098Z   name: test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_41133945556.zip
2025-04-25T05:26:42.5561046Z   retention-days: 14
2025-04-25T05:26:42.5561572Z   if-no-files-found: ignore
2025-04-25T05:26:42.5562228Z   path: test/**/*.xml
test/**/*.csv

2025-04-25T05:26:42.5562872Z   compression-level: 6
2025-04-25T05:26:42.5563398Z   overwrite: false
2025-04-25T05:26:42.5563868Z   include-hidden-files: false
2025-04-25T05:26:42.5564350Z env:
2025-04-25T05:26:42.5564729Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:42.5565471Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:42.5566577Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:42.5567607Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:42.5569345Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:42.5570861Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:42.5571368Z   AWS_REGION: us-east-1
2025-04-25T05:26:42.5572024Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:42.5572838Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:42.5584477Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:42.5585300Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:42.5586134Z ##[endgroup]
2025-04-25T05:26:43.3010170Z With the provided path, there will be 261 files uploaded
2025-04-25T05:26:43.3015686Z Artifact name is valid!
2025-04-25T05:26:43.3016717Z Root directory input is valid!
2025-04-25T05:26:43.4659754Z Beginning upload of artifact content to blob storage
2025-04-25T05:26:43.8664115Z Uploaded bytes 137488
2025-04-25T05:26:43.9156363Z Finished uploading artifact content to blob storage!
2025-04-25T05:26:43.9161767Z SHA256 digest of uploaded artifact zip is 038ca404248cd698b90efa0e31e64ee3db5a09b5500396e32d32a91819c74897
2025-04-25T05:26:43.9164508Z Finalizing artifact upload
2025-04-25T05:26:44.0068089Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_41133945556.zip.zip successfully finalized. Artifact ID 3007209068
2025-04-25T05:26:44.0071148Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_41133945556.zip has been successfully uploaded! Final size is 137488 bytes. Artifact ID is 3007209068
2025-04-25T05:26:44.0081619Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/14656749847/artifacts/3007209068
2025-04-25T05:26:44.0435063Z ##[group]Run actions/upload-artifact@v4
2025-04-25T05:26:44.0435672Z with:
2025-04-25T05:26:44.0436403Z   name: logs-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_41133945556.zip
2025-04-25T05:26:44.0437287Z   retention-days: 14
2025-04-25T05:26:44.0437762Z   if-no-files-found: ignore
2025-04-25T05:26:44.0438295Z   path: usage_log.txt
test/**/*.log

2025-04-25T05:26:44.0438857Z   compression-level: 6
2025-04-25T05:26:44.0439329Z   overwrite: false
2025-04-25T05:26:44.0439972Z   include-hidden-files: false
2025-04-25T05:26:44.0440496Z env:
2025-04-25T05:26:44.0440885Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:44.0441645Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:44.0442804Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:44.0444038Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:44.0446396Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:44.0447986Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:44.0448505Z   AWS_REGION: us-east-1
2025-04-25T05:26:44.0449088Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:44.0449819Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:44.0458479Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:44.0458992Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:44.0459523Z ##[endgroup]
2025-04-25T05:26:44.8175146Z Multiple search paths detected. Calculating the least common ancestor of all paths
2025-04-25T05:26:44.8180211Z The least common ancestor is /var/home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact
2025-04-25T05:26:44.8181684Z With the provided path, there will be 9 files uploaded
2025-04-25T05:26:44.8191238Z Artifact name is valid!
2025-04-25T05:26:44.8192530Z Root directory input is valid!
2025-04-25T05:26:44.9753971Z Beginning upload of artifact content to blob storage
2025-04-25T05:26:45.1904581Z Uploaded bytes 33265
2025-04-25T05:26:45.2304727Z Finished uploading artifact content to blob storage!
2025-04-25T05:26:45.2306652Z SHA256 digest of uploaded artifact zip is 2f6b21b95b15517f3de3540c20e01d46b90ccea2b8fd8d017d90cb260bee8e3b
2025-04-25T05:26:45.2307829Z Finalizing artifact upload
2025-04-25T05:26:45.3206370Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_41133945556.zip.zip successfully finalized. Artifact ID 3007209132
2025-04-25T05:26:45.3209061Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_41133945556.zip has been successfully uploaded! Final size is 33265 bytes. Artifact ID is 3007209132
2025-04-25T05:26:45.3215268Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/14656749847/artifacts/3007209132
2025-04-25T05:26:45.3456652Z ##[group]Run # shellcheck disable=SC2156
2025-04-25T05:26:45.3457516Z # shellcheck disable=SC2156
2025-04-25T05:26:45.3458569Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2025-04-25T05:26:45.3511642Z shell: /usr/bin/bash -e {0}
2025-04-25T05:26:45.3512177Z env:
2025-04-25T05:26:45.3512618Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:45.3513446Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:45.3514652Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:45.3515807Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:45.3518054Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:45.3519833Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:45.3520384Z   AWS_REGION: us-east-1
2025-04-25T05:26:45.3521021Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:45.3521862Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:45.3533115Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:45.3533615Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:45.3534141Z ##[endgroup]
2025-04-25T05:26:45.7099944Z ##[group]Run aws-actions/configure-aws-credentials@v4
2025-04-25T05:26:45.7100715Z with:
2025-04-25T05:26:45.7101520Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_upload-benchmark-results
2025-04-25T05:26:45.7102484Z   role-duration-seconds: 18000
2025-04-25T05:26:45.7103064Z   aws-region: us-east-1
2025-04-25T05:26:45.7103745Z   audience: sts.amazonaws.com
2025-04-25T05:26:45.7104393Z env:
2025-04-25T05:26:45.7104905Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:45.7105924Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:45.7107207Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:45.7108368Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:45.7110314Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:45.7112040Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:45.7112629Z   AWS_REGION: us-east-1
2025-04-25T05:26:45.7113264Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:45.7114020Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:45.7125780Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:45.7126656Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:45.7127594Z ##[endgroup]
2025-04-25T05:26:46.0339070Z Assuming role with OIDC
2025-04-25T05:26:46.2577309Z Authenticated as assumedRoleId AROAUPVRELQNA5GQHA6IA:GitHubActions
2025-04-25T05:26:46.3397851Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main
2025-04-25T05:26:46.3398850Z with:
2025-04-25T05:26:46.3399392Z   benchmark-results-dir: test/test-reports
2025-04-25T05:26:46.3400471Z   dry-run: false
2025-04-25T05:26:46.3400989Z   schema-version: v3
2025-04-25T05:26:46.3401984Z   github-token: ***
2025-04-25T05:26:46.3402630Z env:
2025-04-25T05:26:46.3403193Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:46.3404213Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:46.3405316Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:46.3406306Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:46.3407952Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:46.3409453Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:46.3409978Z   AWS_REGION: us-east-1
2025-04-25T05:26:46.3410505Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:46.3411164Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:46.3420817Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:46.3421567Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:46.3422364Z ##[endgroup]
2025-04-25T05:26:46.3455682Z ##[group]Run set -eux
2025-04-25T05:26:46.3456159Z set -eux
2025-04-25T05:26:46.3456646Z python3 -mpip install boto3==1.35.33
2025-04-25T05:26:46.3505836Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:46.3506514Z env:
2025-04-25T05:26:46.3506910Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:46.3508083Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:46.3509175Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:46.3510183Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:46.3511820Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:46.3513267Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:46.3513796Z   AWS_REGION: us-east-1
2025-04-25T05:26:46.3514332Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:46.3514991Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:46.3524514Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:46.3525296Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:46.3526097Z ##[endgroup]
2025-04-25T05:26:46.3610800Z + python3 -mpip install boto3==1.35.33
2025-04-25T05:26:46.6463978Z Defaulting to user installation because normal site-packages is not writeable
2025-04-25T05:26:46.8128489Z Requirement already satisfied: boto3==1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (1.35.33)
2025-04-25T05:26:46.8162054Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (1.35.99)
2025-04-25T05:26:46.8165553Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3/dist-packages (from boto3==1.35.33) (0.10.0)
2025-04-25T05:26:46.8170473Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (0.10.4)
2025-04-25T05:26:46.8217295Z Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.26.5)
2025-04-25T05:26:46.8221419Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.9.0.post0)
2025-04-25T05:26:46.8257485Z Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.16.0)
2025-04-25T05:26:46.9917258Z ##[group]Run set -eux
2025-04-25T05:26:46.9917859Z set -eux
2025-04-25T05:26:46.9918306Z 
2025-04-25T05:26:46.9918806Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2025-04-25T05:26:46.9919668Z   echo "Missing github-token input"
2025-04-25T05:26:46.9920278Z   exit 1
2025-04-25T05:26:46.9920718Z fi
2025-04-25T05:26:46.9971782Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:46.9972173Z env:
2025-04-25T05:26:46.9972415Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:46.9972865Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:46.9973529Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:46.9974119Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:46.9975063Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:46.9975896Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:46.9976197Z   AWS_REGION: us-east-1
2025-04-25T05:26:46.9976553Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:46.9976954Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:46.9987570Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:46.9988434Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:46.9989587Z   GITHUB_TOKEN: ***
2025-04-25T05:26:46.9990089Z ##[endgroup]
2025-04-25T05:26:47.0081170Z + [[ -z *** ]]
2025-04-25T05:26:47.0153654Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2025-04-25T05:26:47.0154926Z with:
2025-04-25T05:26:47.0155629Z   github-token: ***
2025-04-25T05:26:47.0156123Z env:
2025-04-25T05:26:47.0156569Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:47.0157427Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:47.0158626Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:47.0159862Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:47.0161709Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:47.0163354Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:47.0163922Z   AWS_REGION: us-east-1
2025-04-25T05:26:47.0164511Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:47.0165253Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:47.0174183Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:47.0174627Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:47.0175097Z ##[endgroup]
2025-04-25T05:26:47.0204802Z ##[group]Run set -eux
2025-04-25T05:26:47.0205354Z set -eux
2025-04-25T05:26:47.0205822Z 
2025-04-25T05:26:47.0206843Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-04-25T05:26:47.0253085Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:47.0253534Z env:
2025-04-25T05:26:47.0253822Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:47.0254351Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:47.0255069Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:47.0255756Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:47.0256897Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:47.0257891Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:47.0258590Z   AWS_REGION: us-east-1
2025-04-25T05:26:47.0259118Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:47.0259989Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:47.0270680Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:47.0271539Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:47.0272646Z   GITHUB_TOKEN: ***
2025-04-25T05:26:47.0273127Z ##[endgroup]
2025-04-25T05:26:47.0369438Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 14656749847 gpu5966
2025-04-25T05:26:47.9928357Z setting job-id=41133945556
2025-04-25T05:26:47.9929374Z setting job-name=linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T05:26:48.0152071Z ##[group]Run set -eux
2025-04-25T05:26:48.0152663Z set -eux
2025-04-25T05:26:48.0153169Z 
2025-04-25T05:26:48.0154011Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2025-04-25T05:26:48.0155070Z   --schema-version "${SCHEMA_VERSION}" \
2025-04-25T05:26:48.0155806Z   --repo "${REPO}" \
2025-04-25T05:26:48.0156480Z   --head-branch "${HEAD_BRANCH}" \
2025-04-25T05:26:48.0157179Z   --head-sha "${HEAD_SHA}" \
2025-04-25T05:26:48.0157882Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2025-04-25T05:26:48.0158614Z   --run-attempt "${RUN_ATTEMPT}" \
2025-04-25T05:26:48.0159284Z   --job-id "${JOB_ID}" \
2025-04-25T05:26:48.0160089Z   --job-name "${JOB_NAME}"
2025-04-25T05:26:48.0216938Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:48.0217721Z env:
2025-04-25T05:26:48.0218210Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:48.0219583Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:48.0220888Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:48.0222131Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:48.0224066Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:48.0225772Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:48.0226390Z   AWS_REGION: us-east-1
2025-04-25T05:26:48.0227060Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:48.0227857Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:48.0238996Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:48.0240093Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:48.0241062Z   SCHEMA_VERSION: v3
2025-04-25T05:26:48.0241632Z   REPO: pytorch/pytorch
2025-04-25T05:26:48.0242240Z   HEAD_BRANCH: refs/tags/ciflow/trunk/148893
2025-04-25T05:26:48.0243000Z   HEAD_SHA: b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-25T05:26:48.0243747Z   WORKFLOW_RUN_ID: 14656749847
2025-04-25T05:26:48.0244337Z   RUN_ATTEMPT: 1
2025-04-25T05:26:48.0244833Z   JOB_ID: 41133945556
2025-04-25T05:26:48.0245649Z   JOB_NAME: linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-04-25T05:26:48.0246552Z ##[endgroup]
2025-04-25T05:26:48.0348078Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/tags/ciflow/trunk/148893 --head-sha b68c0effe97351b0c24b2f7aef671559be6aa977 --workflow-id 14656749847 --run-attempt 1 --job-id 41133945556 --job-name 'linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)'
2025-04-25T05:26:48.0822463Z ##[group]Run set -eux
2025-04-25T05:26:48.0823032Z set -eux
2025-04-25T05:26:48.0823477Z 
2025-04-25T05:26:48.0823981Z # TODO (huydhn): Implement this part
2025-04-25T05:26:48.0825097Z echo "runners=[]" >> "${GITHUB_OUTPUT}"
2025-04-25T05:26:48.0879928Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:48.0880714Z env:
2025-04-25T05:26:48.0881171Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:48.0882020Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:48.0883227Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:48.0884339Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:48.0886154Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:48.0888056Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:48.0888732Z   AWS_REGION: us-east-1
2025-04-25T05:26:48.0889462Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:48.0890356Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:48.0901726Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:48.0902568Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:48.0903452Z ##[endgroup]
2025-04-25T05:26:48.0994623Z + echo 'runners=[]'
2025-04-25T05:26:48.1060463Z ##[group]Run set -eux
2025-04-25T05:26:48.1061059Z set -eux
2025-04-25T05:26:48.1061556Z 
2025-04-25T05:26:48.1062127Z # TODO (huydhn): Implement this part
2025-04-25T05:26:48.1062939Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2025-04-25T05:26:48.1117447Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:48.1118249Z env:
2025-04-25T05:26:48.1118744Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:48.1119773Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:48.1121606Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:48.1122814Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:48.1124745Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:48.1126476Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:48.1127146Z   AWS_REGION: us-east-1
2025-04-25T05:26:48.1127911Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:48.1128856Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:48.1136406Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:48.1136835Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:48.1137296Z ##[endgroup]
2025-04-25T05:26:48.1224262Z + echo 'dependencies={}'
2025-04-25T05:26:48.1271241Z ##[group]Run set -eux
2025-04-25T05:26:48.1271836Z set -eux
2025-04-25T05:26:48.1272287Z 
2025-04-25T05:26:48.1272822Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2025-04-25T05:26:48.1273691Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2025-04-25T05:26:48.1274614Z   # We don't want the job to fail if the directory doesn't exist
2025-04-25T05:26:48.1275365Z   exit 0
2025-04-25T05:26:48.1275817Z fi
2025-04-25T05:26:48.1276235Z 
2025-04-25T05:26:48.1276711Z if [[ "${DRY_RUN}" == "true" ]]; then
2025-04-25T05:26:48.1277617Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-04-25T05:26:48.1278680Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-04-25T05:26:48.1279666Z     --metadata "${BENCHMARK_METADATA}" \
2025-04-25T05:26:48.1280370Z     --runners "${RUNNER_INFO}" \
2025-04-25T05:26:48.1281047Z     --dependencies "${DEPENDENCIES}" \
2025-04-25T05:26:48.1281702Z     --dry-run
2025-04-25T05:26:48.1282182Z else
2025-04-25T05:26:48.1282910Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-04-25T05:26:48.1284333Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-04-25T05:26:48.1285175Z     --metadata "${BENCHMARK_METADATA}" \
2025-04-25T05:26:48.1285856Z     --runners "${RUNNER_INFO}" \
2025-04-25T05:26:48.1286518Z     --dependencies "${DEPENDENCIES}"
2025-04-25T05:26:48.1287126Z fi
2025-04-25T05:26:48.1342706Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:48.1343447Z env:
2025-04-25T05:26:48.1343907Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:48.1344758Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:48.1345949Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:48.1347068Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:48.1348896Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:48.1350513Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:48.1351089Z   AWS_REGION: us-east-1
2025-04-25T05:26:48.1351711Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:48.1352464Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:48.1363137Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:48.1363993Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:48.1364943Z   BENCHMARK_RESULTS_DIR: test/test-reports
2025-04-25T05:26:48.1365569Z   DRY_RUN: false
2025-04-25T05:26:48.1367922Z   BENCHMARK_METADATA: {"timestamp": 1745558808, "schema_version": "v3", "name": "linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)", "repo": "pytorch/pytorch", "head_branch": "refs/tags/ciflow/trunk/148893", "head_sha": "b68c0effe97351b0c24b2f7aef671559be6aa977", "workflow_id": 14656749847, "run_attempt": 1, "job_id": 41133945556}
2025-04-25T05:26:48.1370754Z   RUNNER_INFO: []
2025-04-25T05:26:48.1371240Z   DEPENDENCIES: {}
2025-04-25T05:26:48.1371719Z ##[endgroup]
2025-04-25T05:26:48.1468555Z + [[ ! -d test/test-reports ]]
2025-04-25T05:26:48.1469119Z + [[ false == \t\r\u\e ]]
2025-04-25T05:26:48.1472860Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1745558808, "schema_version": "v3", "name": "linux-focal-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)", "repo": "pytorch/pytorch", "head_branch": "refs/tags/ciflow/trunk/148893", "head_sha": "b68c0effe97351b0c24b2f7aef671559be6aa977", "workflow_id": 14656749847, "run_attempt": 1, "job_id": 41133945556}' --runners '[]' --dependencies '{}'
2025-04-25T05:26:48.3037331Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'distributed/test_c10d_common'}, {'test_file': 'distributed/test_c10d_nccl'}], 'excluded': []} from test/test-reports/td_exclusions-cc4ca970b7754ed24a5c.json is not a benchmark record, skipping
2025-04-25T05:26:48.3040864Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-04-25T05:26:48.3043817Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/test_cpp_rpc'}], 'excluded': []} from test/test-reports/td_exclusions-020b48e800b3067e1f03.json is not a benchmark record, skipping
2025-04-25T05:26:48.3046687Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-04-25T05:26:48.3050086Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/HashStoreTest'}], 'excluded': []} from test/test-reports/td_exclusions-549ae6f890614bd7ca5a.json is not a benchmark record, skipping
2025-04-25T05:26:48.3052973Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-04-25T05:26:48.3055806Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/TCPStoreTest'}], 'excluded': []} from test/test-reports/td_exclusions-efbef9ea8e13d2c20dbf.json is not a benchmark record, skipping
2025-04-25T05:26:48.3058647Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-04-25T05:26:48.3061398Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'test_linalg'}], 'excluded': []} from test/test-reports/td_exclusions-239d4214da82c44696b9.json is not a benchmark record, skipping
2025-04-25T05:26:48.3064170Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-04-25T05:26:48.3066994Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/FileStoreTest'}], 'excluded': []} from test/test-reports/td_exclusions-54dfed906b159bb19df3.json is not a benchmark record, skipping
2025-04-25T05:26:48.3069792Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-04-25T05:26:48.3274941Z Prepare all required actions
2025-04-25T05:26:48.3275810Z Getting action download info
2025-04-25T05:26:48.3334218Z ##[group]Run ./.github/actions/teardown-rocm
2025-04-25T05:26:48.3335503Z env:
2025-04-25T05:26:48.3336071Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:48.3337113Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:48.3338364Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:48.3339536Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:48.3341477Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:48.3343312Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:48.3344040Z   AWS_REGION: us-east-1
2025-04-25T05:26:48.3344817Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:48.3345794Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:48.3356926Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:48.3357824Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:48.3358792Z ##[endgroup]
2025-04-25T05:26:48.3393367Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-04-25T05:26:48.3394453Z # ignore expansion of "docker ps -q" since it could be empty
2025-04-25T05:26:48.3395294Z # shellcheck disable=SC2046
2025-04-25T05:26:48.3396003Z docker stop $(docker ps -q) || true
2025-04-25T05:26:48.3396709Z # Prune all stopped containers.
2025-04-25T05:26:48.3397370Z docker container prune -f
2025-04-25T05:26:48.3444771Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:48.3445447Z env:
2025-04-25T05:26:48.3445854Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:48.3446599Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:48.3447668Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:48.3448666Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:48.3450295Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:48.3451708Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:48.3452214Z   AWS_REGION: us-east-1
2025-04-25T05:26:48.3452758Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:48.3453394Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:48.3462629Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:48.3463375Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:48.3464160Z ##[endgroup]
2025-04-25T05:26:49.0534953Z e52d4a5a5094
2025-04-25T05:26:55.6772288Z Deleted Containers:
2025-04-25T05:26:55.6772801Z e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:55.6773134Z 
2025-04-25T05:26:55.6773266Z Total reclaimed space: 7.979GB
2025-04-25T05:26:55.6864473Z Prepare all required actions
2025-04-25T05:26:55.6888937Z ##[group]Run ./.github/actions/diskspace-cleanup
2025-04-25T05:26:55.6889208Z with:
2025-04-25T05:26:55.6889407Z   diskspace-cutoff: 70
2025-04-25T05:26:55.6889628Z env:
2025-04-25T05:26:55.6889821Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:55.6890169Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:55.6890661Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:55.6891119Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:55.6891868Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:55.6892796Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:55.6893060Z   AWS_REGION: us-east-1
2025-04-25T05:26:55.6893337Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:55.6893861Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:55.6898131Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:55.6898492Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:55.6898870Z ##[endgroup]
2025-04-25T05:26:55.6912014Z ##[group]Run set -ex
2025-04-25T05:26:55.6912250Z set -ex
2025-04-25T05:26:55.6912457Z diskspace_cutoff=70
2025-04-25T05:26:55.6912753Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-04-25T05:26:55.6913090Z if [ ! -d "$docker_root_dir" ]; then
2025-04-25T05:26:55.6913506Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-04-25T05:26:55.6913888Z     exit 0
2025-04-25T05:26:55.6914085Z fi
2025-04-25T05:26:55.6914429Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-04-25T05:26:55.6915110Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-04-25T05:26:55.6915699Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-04-25T05:26:55.6916006Z     docker system prune -af
2025-04-25T05:26:55.6916406Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-04-25T05:26:55.6916866Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-04-25T05:26:55.6917329Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2025-04-25T05:26:55.6917741Z         echo "$msg"
2025-04-25T05:26:55.6917969Z         exit 1
2025-04-25T05:26:55.6918172Z     else
2025-04-25T05:26:55.6918419Z         difference=$((diskspace - diskspace_new))
2025-04-25T05:26:55.6918741Z         echo "Diskspace saved: $difference percent"
2025-04-25T05:26:55.6919018Z     fi
2025-04-25T05:26:55.6919208Z fi
2025-04-25T05:26:55.6970140Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-25T05:26:55.6971024Z env:
2025-04-25T05:26:55.6971573Z   GIT_DEFAULT_BRANCH: main
2025-04-25T05:26:55.6972570Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-04-25T05:26:55.6973973Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-04-25T05:26:55.6975316Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-04-25T05:26:55.6977165Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-04-25T05:26:55.6978798Z   AWS_DEFAULT_REGION: us-east-1
2025-04-25T05:26:55.6979384Z   AWS_REGION: us-east-1
2025-04-25T05:26:55.6979992Z   AWS_ACCESS_KEY_ID: ***
2025-04-25T05:26:55.6980747Z   AWS_SECRET_ACCESS_KEY: ***
2025-04-25T05:26:55.6991832Z   AWS_SESSION_TOKEN: ***
2025-04-25T05:26:55.6992687Z   CONTAINER_NAME: e52d4a5a50948e9003f4f8ddf853a7c4252842baec9aa458f7648bad2c69f1e6
2025-04-25T05:26:55.6993597Z ##[endgroup]
2025-04-25T05:26:55.7065592Z + diskspace_cutoff=70
2025-04-25T05:26:55.7076679Z ++ docker info -f '{{.DockerRootDir}}'
2025-04-25T05:26:55.8091943Z + docker_root_dir=/media/4TB/docker-rootless
2025-04-25T05:26:55.8092748Z + '[' '!' -d /media/4TB/docker-rootless ']'
2025-04-25T05:26:55.8108155Z ++ df -H --output=pcent /media/4TB/docker-rootless
2025-04-25T05:26:55.8111707Z ++ sed -n 2p
2025-04-25T05:26:55.8114758Z ++ sed s/%//
2025-04-25T05:26:55.8116372Z ++ sed 's/ //'
2025-04-25T05:26:55.8161894Z + diskspace=20
2025-04-25T05:26:55.8163948Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-04-25T05:26:55.8165353Z + [[ 20 -ge 70 ]]
2025-04-25T05:26:55.8260020Z Post job cleanup.
2025-04-25T05:26:55.8386195Z Post job cleanup.
2025-04-25T05:26:55.9811414Z Post job cleanup.
2025-04-25T05:26:56.0485126Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-25T05:26:56.1186501Z Post job cleanup.
2025-04-25T05:26:56.2669235Z Post job cleanup.
2025-04-25T05:26:56.2754547Z Post job cleanup.
2025-04-25T05:26:56.3904748Z [command]/usr/bin/git version
2025-04-25T05:26:56.3963053Z git version 2.34.1
2025-04-25T05:26:56.3996718Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/bd0be134-8574-4d33-b14c-8b8579d549c0/.gitconfig'
2025-04-25T05:26:56.4010382Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/bd0be134-8574-4d33-b14c-8b8579d549c0' before making global git config changes
2025-04-25T05:26:56.4012212Z Adding repository directory to the temporary git global config as a safe directory
2025-04-25T05:26:56.4014765Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-04-25T05:26:56.4097582Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-04-25T05:26:56.4173601Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-04-25T05:26:56.4858559Z Entering 'android/libs/fbjni'
2025-04-25T05:26:56.4991196Z Entering 'third_party/FP16'
2025-04-25T05:26:56.5115727Z Entering 'third_party/FXdiv'
2025-04-25T05:26:56.5238314Z Entering 'third_party/NNPACK'
2025-04-25T05:26:56.5360169Z Entering 'third_party/NVTX'
2025-04-25T05:26:56.5486928Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-25T05:26:56.5604540Z Entering 'third_party/XNNPACK'
2025-04-25T05:26:56.5765251Z Entering 'third_party/benchmark'
2025-04-25T05:26:56.5888162Z Entering 'third_party/composable_kernel'
2025-04-25T05:26:56.6024990Z Entering 'third_party/cpp-httplib'
2025-04-25T05:26:56.6145018Z Entering 'third_party/cpuinfo'
2025-04-25T05:26:56.6273320Z Entering 'third_party/cudnn_frontend'
2025-04-25T05:26:56.6404019Z Entering 'third_party/cutlass'
2025-04-25T05:26:56.6552037Z Entering 'third_party/eigen'
2025-04-25T05:26:56.6681477Z Entering 'third_party/fbgemm'
2025-04-25T05:26:56.6809104Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-25T05:26:56.6927818Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T05:26:56.7045920Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-25T05:26:56.7184091Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-25T05:26:56.7304710Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T05:26:56.7429283Z Entering 'third_party/flash-attention'
2025-04-25T05:26:56.7549716Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T05:26:56.7682683Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-25T05:26:56.7826664Z Entering 'third_party/flatbuffers'
2025-04-25T05:26:56.7953290Z Entering 'third_party/fmt'
2025-04-25T05:26:56.8075455Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-25T05:26:56.8195542Z Entering 'third_party/gloo'
2025-04-25T05:26:56.8325943Z Entering 'third_party/googletest'
2025-04-25T05:26:56.8444811Z Entering 'third_party/ideep'
2025-04-25T05:26:56.8568048Z Entering 'third_party/ideep/mkl-dnn'
2025-04-25T05:26:56.8708707Z Entering 'third_party/ittapi'
2025-04-25T05:26:56.8830221Z Entering 'third_party/kineto'
2025-04-25T05:26:56.8946140Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T05:26:56.9063801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T05:26:56.9184817Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T05:26:56.9304872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T05:26:56.9424388Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T05:26:56.9539962Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T05:26:56.9668024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T05:26:56.9786672Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T05:26:56.9908350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T05:26:57.0029782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T05:26:57.0155037Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T05:26:57.0275608Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T05:26:57.0403338Z Entering 'third_party/kleidiai'
2025-04-25T05:26:57.0524438Z Entering 'third_party/mimalloc'
2025-04-25T05:26:57.0642494Z Entering 'third_party/nlohmann'
2025-04-25T05:26:57.0763833Z Entering 'third_party/onnx'
2025-04-25T05:26:57.0915262Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-25T05:26:57.1047714Z Entering 'third_party/opentelemetry-cpp'
2025-04-25T05:26:57.1168985Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T05:26:57.1288173Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T05:26:57.1406350Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T05:26:57.1522158Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T05:26:57.1642443Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T05:26:57.1758537Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T05:26:57.1875580Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T05:26:57.1990016Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T05:26:57.2114881Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T05:26:57.2241374Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T05:26:57.2408073Z Entering 'third_party/pocketfft'
2025-04-25T05:26:57.2530800Z Entering 'third_party/protobuf'
2025-04-25T05:26:57.2652380Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-25T05:26:57.2771676Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-25T05:26:57.2898436Z Entering 'third_party/psimd'
2025-04-25T05:26:57.3021921Z Entering 'third_party/pthreadpool'
2025-04-25T05:26:57.3143757Z Entering 'third_party/pybind11'
2025-04-25T05:26:57.3267400Z Entering 'third_party/python-peachpy'
2025-04-25T05:26:57.3390554Z Entering 'third_party/sleef'
2025-04-25T05:26:57.3511509Z Entering 'third_party/tensorpipe'
2025-04-25T05:26:57.3635429Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-25T05:26:57.3756021Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-25T05:26:57.3872949Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-25T05:26:57.3993729Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T05:26:57.4107718Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T05:26:57.4300569Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-04-25T05:26:57.4340597Z http.https://github.com/.extraheader
2025-04-25T05:26:57.4364454Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-04-25T05:26:57.4448088Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-04-25T05:26:57.5120102Z Entering 'android/libs/fbjni'
2025-04-25T05:26:57.5189838Z http.https://github.com/.extraheader
2025-04-25T05:26:57.5289331Z Entering 'third_party/FP16'
2025-04-25T05:26:57.5355539Z http.https://github.com/.extraheader
2025-04-25T05:26:57.5451785Z Entering 'third_party/FXdiv'
2025-04-25T05:26:57.5517554Z http.https://github.com/.extraheader
2025-04-25T05:26:57.5616645Z Entering 'third_party/NNPACK'
2025-04-25T05:26:57.5680662Z http.https://github.com/.extraheader
2025-04-25T05:26:57.5779835Z Entering 'third_party/NVTX'
2025-04-25T05:26:57.5846189Z http.https://github.com/.extraheader
2025-04-25T05:26:57.5944338Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-25T05:26:57.6012276Z http.https://github.com/.extraheader
2025-04-25T05:26:57.6111312Z Entering 'third_party/XNNPACK'
2025-04-25T05:26:57.6177126Z http.https://github.com/.extraheader
2025-04-25T05:26:57.6304963Z Entering 'third_party/benchmark'
2025-04-25T05:26:57.6368995Z http.https://github.com/.extraheader
2025-04-25T05:26:57.6465538Z Entering 'third_party/composable_kernel'
2025-04-25T05:26:57.6530617Z http.https://github.com/.extraheader
2025-04-25T05:26:57.6640127Z Entering 'third_party/cpp-httplib'
2025-04-25T05:26:57.6705916Z http.https://github.com/.extraheader
2025-04-25T05:26:57.6802661Z Entering 'third_party/cpuinfo'
2025-04-25T05:26:57.6872667Z http.https://github.com/.extraheader
2025-04-25T05:26:57.6970083Z Entering 'third_party/cudnn_frontend'
2025-04-25T05:26:57.7034442Z http.https://github.com/.extraheader
2025-04-25T05:26:57.7131108Z Entering 'third_party/cutlass'
2025-04-25T05:26:57.7196207Z http.https://github.com/.extraheader
2025-04-25T05:26:57.7309641Z Entering 'third_party/eigen'
2025-04-25T05:26:57.7373451Z http.https://github.com/.extraheader
2025-04-25T05:26:57.7471842Z Entering 'third_party/fbgemm'
2025-04-25T05:26:57.7537392Z http.https://github.com/.extraheader
2025-04-25T05:26:57.7628737Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-25T05:26:57.7691541Z http.https://github.com/.extraheader
2025-04-25T05:26:57.7784241Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-25T05:26:57.7847405Z http.https://github.com/.extraheader
2025-04-25T05:26:57.7945852Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-25T05:26:57.8008653Z http.https://github.com/.extraheader
2025-04-25T05:26:57.8116895Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-25T05:26:57.8179266Z http.https://github.com/.extraheader
2025-04-25T05:26:57.8273496Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-25T05:26:57.8336893Z http.https://github.com/.extraheader
2025-04-25T05:26:57.8438643Z Entering 'third_party/flash-attention'
2025-04-25T05:26:57.8503602Z http.https://github.com/.extraheader
2025-04-25T05:26:57.8592319Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-25T05:26:57.8655946Z http.https://github.com/.extraheader
2025-04-25T05:26:57.8772869Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-25T05:26:57.8835913Z http.https://github.com/.extraheader
2025-04-25T05:26:57.8960170Z Entering 'third_party/flatbuffers'
2025-04-25T05:26:57.9027685Z http.https://github.com/.extraheader
2025-04-25T05:26:57.9128855Z Entering 'third_party/fmt'
2025-04-25T05:26:57.9194176Z http.https://github.com/.extraheader
2025-04-25T05:26:57.9288221Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-25T05:26:57.9355233Z http.https://github.com/.extraheader
2025-04-25T05:26:57.9452621Z Entering 'third_party/gloo'
2025-04-25T05:26:57.9517773Z http.https://github.com/.extraheader
2025-04-25T05:26:57.9612348Z Entering 'third_party/googletest'
2025-04-25T05:26:57.9677270Z http.https://github.com/.extraheader
2025-04-25T05:26:57.9772914Z Entering 'third_party/ideep'
2025-04-25T05:26:57.9838339Z http.https://github.com/.extraheader
2025-04-25T05:26:57.9926507Z Entering 'third_party/ideep/mkl-dnn'
2025-04-25T05:26:57.9989598Z http.https://github.com/.extraheader
2025-04-25T05:26:58.0105387Z Entering 'third_party/ittapi'
2025-04-25T05:26:58.0171501Z http.https://github.com/.extraheader
2025-04-25T05:26:58.0265682Z Entering 'third_party/kineto'
2025-04-25T05:26:58.0333639Z http.https://github.com/.extraheader
2025-04-25T05:26:58.0426384Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-25T05:26:58.0490172Z http.https://github.com/.extraheader
2025-04-25T05:26:58.0584912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-25T05:26:58.0649654Z http.https://github.com/.extraheader
2025-04-25T05:26:58.0750517Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-25T05:26:58.0813757Z http.https://github.com/.extraheader
2025-04-25T05:26:58.0912205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-25T05:26:58.0974665Z http.https://github.com/.extraheader
2025-04-25T05:26:58.1072601Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-25T05:26:58.1137570Z http.https://github.com/.extraheader
2025-04-25T05:26:58.1229301Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-25T05:26:58.1294159Z http.https://github.com/.extraheader
2025-04-25T05:26:58.1398441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-25T05:26:58.1462504Z http.https://github.com/.extraheader
2025-04-25T05:26:58.1560677Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-25T05:26:58.1624639Z http.https://github.com/.extraheader
2025-04-25T05:26:58.1717925Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-25T05:26:58.1780520Z http.https://github.com/.extraheader
2025-04-25T05:26:58.1882919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-25T05:26:58.1943022Z http.https://github.com/.extraheader
2025-04-25T05:26:58.2044802Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-25T05:26:58.2106885Z http.https://github.com/.extraheader
2025-04-25T05:26:58.2202764Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-25T05:26:58.2267551Z http.https://github.com/.extraheader
2025-04-25T05:26:58.2368412Z Entering 'third_party/kleidiai'
2025-04-25T05:26:58.2433967Z http.https://github.com/.extraheader
2025-04-25T05:26:58.2533749Z Entering 'third_party/mimalloc'
2025-04-25T05:26:58.2598783Z http.https://github.com/.extraheader
2025-04-25T05:26:58.2691993Z Entering 'third_party/nlohmann'
2025-04-25T05:26:58.2761820Z http.https://github.com/.extraheader
2025-04-25T05:26:58.2863317Z Entering 'third_party/onnx'
2025-04-25T05:26:58.2927733Z http.https://github.com/.extraheader
2025-04-25T05:26:58.3057340Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-25T05:26:58.3122134Z http.https://github.com/.extraheader
2025-04-25T05:26:58.3229704Z Entering 'third_party/opentelemetry-cpp'
2025-04-25T05:26:58.3294728Z http.https://github.com/.extraheader
2025-04-25T05:26:58.3389218Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-25T05:26:58.3452144Z http.https://github.com/.extraheader
2025-04-25T05:26:58.3545698Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-25T05:26:58.3609133Z http.https://github.com/.extraheader
2025-04-25T05:26:58.3708440Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-25T05:26:58.3773026Z http.https://github.com/.extraheader
2025-04-25T05:26:58.3863819Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-25T05:26:58.3926631Z http.https://github.com/.extraheader
2025-04-25T05:26:58.4023163Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-25T05:26:58.4086557Z http.https://github.com/.extraheader
2025-04-25T05:26:58.4179592Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-25T05:26:58.4244173Z http.https://github.com/.extraheader
2025-04-25T05:26:58.4337342Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-25T05:26:58.4402024Z http.https://github.com/.extraheader
2025-04-25T05:26:58.4492352Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-25T05:26:58.4554560Z http.https://github.com/.extraheader
2025-04-25T05:26:58.4653939Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-25T05:26:58.4717431Z http.https://github.com/.extraheader
2025-04-25T05:26:58.4819680Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-25T05:26:58.4883493Z http.https://github.com/.extraheader
2025-04-25T05:26:58.5028669Z Entering 'third_party/pocketfft'
2025-04-25T05:26:58.5093719Z http.https://github.com/.extraheader
2025-04-25T05:26:58.5191426Z Entering 'third_party/protobuf'
2025-04-25T05:26:58.5258412Z http.https://github.com/.extraheader
2025-04-25T05:26:58.5355204Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-25T05:26:58.5419654Z http.https://github.com/.extraheader
2025-04-25T05:26:58.5513433Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-25T05:26:58.5576442Z http.https://github.com/.extraheader
2025-04-25T05:26:58.5677406Z Entering 'third_party/psimd'
2025-04-25T05:26:58.5743203Z http.https://github.com/.extraheader
2025-04-25T05:26:58.5840358Z Entering 'third_party/pthreadpool'
2025-04-25T05:26:58.5907996Z http.https://github.com/.extraheader
2025-04-25T05:26:58.6007923Z Entering 'third_party/pybind11'
2025-04-25T05:26:58.6073296Z http.https://github.com/.extraheader
2025-04-25T05:26:58.6167733Z Entering 'third_party/python-peachpy'
2025-04-25T05:26:58.6233509Z http.https://github.com/.extraheader
2025-04-25T05:26:58.6324279Z Entering 'third_party/sleef'
2025-04-25T05:26:58.6388883Z http.https://github.com/.extraheader
2025-04-25T05:26:58.6484766Z Entering 'third_party/tensorpipe'
2025-04-25T05:26:58.6549520Z http.https://github.com/.extraheader
2025-04-25T05:26:58.6643708Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-25T05:26:58.6705268Z http.https://github.com/.extraheader
2025-04-25T05:26:58.6799630Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-25T05:26:58.6863398Z http.https://github.com/.extraheader
2025-04-25T05:26:58.6955724Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-25T05:26:58.7019145Z http.https://github.com/.extraheader
2025-04-25T05:26:58.7112999Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-25T05:26:58.7176267Z http.https://github.com/.extraheader
2025-04-25T05:26:58.7264863Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-25T05:26:58.7329044Z http.https://github.com/.extraheader
2025-04-25T05:26:58.7745545Z Cleaning up orphan processes