2025-03-14T04:29:51.2711705Z Current runner version: '2.322.0' 2025-03-14T04:29:51.2720407Z Runner name: 'i-0f8cc103977fc9a05' 2025-03-14T04:29:51.2721651Z Runner group name: 'Default' 2025-03-14T04:29:51.2722762Z Machine name: 'ip-10-1-65-155' 2025-03-14T04:29:51.2728028Z ##[group]GITHUB_TOKEN Permissions 2025-03-14T04:29:51.2730886Z Actions: read 2025-03-14T04:29:51.2731636Z Attestations: read 2025-03-14T04:29:51.2732467Z Checks: read 2025-03-14T04:29:51.2733129Z Contents: read 2025-03-14T04:29:51.2733754Z Deployments: read 2025-03-14T04:29:51.2734479Z Discussions: read 2025-03-14T04:29:51.2735158Z Issues: read 2025-03-14T04:29:51.2735788Z Metadata: read 2025-03-14T04:29:51.2736517Z Packages: read 2025-03-14T04:29:51.2737178Z Pages: read 2025-03-14T04:29:51.2737865Z PullRequests: read 2025-03-14T04:29:51.2738600Z RepositoryProjects: read 2025-03-14T04:29:51.2739350Z SecurityEvents: read 2025-03-14T04:29:51.2740177Z Statuses: read 2025-03-14T04:29:51.2740849Z ##[endgroup] 2025-03-14T04:29:51.2744515Z Secret source: Actions 2025-03-14T04:29:51.2745528Z Prepare workflow directory 2025-03-14T04:29:51.6104696Z Prepare all required actions 2025-03-14T04:29:51.6150451Z Getting action download info 2025-03-14T04:29:51.7915159Z Download action repository 'pytorch/test-infra@main' (SHA:de00dac6adc071cb2f9861380a0ed3947b93e5cc) 2025-03-14T04:29:53.3094680Z Download action repository 'pytorch/pytorch@main' (SHA:bea181ff7eeead9fcdd806e286846296c4ab2d67) 2025-03-14T04:30:07.4489369Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2025-03-14T04:30:07.6437187Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-03-14T04:30:07.9211620Z Getting action download info 2025-03-14T04:30:08.0264546Z Download action repository 'actions/checkout@v4' (SHA:11bd71901bbe5b1630ceea73d27597364c9af683) 2025-03-14T04:30:08.2676547Z Getting action download info 2025-03-14T04:30:08.3711713Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-03-14T04:30:08.5556822Z Getting action download info 2025-03-14T04:30:08.6578705Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-03-14T04:30:08.8065157Z Getting action download info 2025-03-14T04:30:08.9365403Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (aed0b7a742a2d7b7901790622829cbd2135049a4) 2025-03-14T04:30:08.9367700Z ##[group] Inputs 2025-03-14T04:30:08.9368123Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2025-03-14T04:30:08.9370049Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}]} 2025-03-14T04:30:08.9372101Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:08.9373049Z sync-tag: 2025-03-14T04:30:08.9373921Z timeout-minutes: 360 2025-03-14T04:30:08.9374242Z use-gha: 2025-03-14T04:30:08.9374521Z dashboard-tag: 2025-03-14T04:30:08.9374831Z s3-bucket: gha-artifacts 2025-03-14T04:30:08.9375174Z aws-role-to-assume: 2025-03-14T04:30:08.9375834Z disable-monitor: false 2025-03-14T04:30:08.9376193Z ##[endgroup] 2025-03-14T04:30:08.9376815Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T04:30:09.0058798Z A job started hook has been configured by the self-hosted runner administrator 2025-03-14T04:30:09.0166832Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-03-14T04:30:09.0177380Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:09.0178153Z ##[endgroup] 2025-03-14T04:30:10.6299409Z Runner Type: lf.linux.g4dn.12xlarge.nvidia.gpu 2025-03-14T04:30:10.6300538Z Instance Type: g4dn.12xlarge 2025-03-14T04:30:10.6300910Z AMI Name: unknown 2025-03-14T04:30:10.6330643Z AMI ID: ami-05b10e08d247fb927 2025-03-14T04:30:16.1898604Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-03-14T04:30:16.1899104Z with: 2025-03-14T04:30:16.1899780Z github-secret: *** 2025-03-14T04:30:16.1900568Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-03-14T04:30:16.1901405Z activate-with-label: false 2025-03-14T04:30:16.1901727Z label: with-ssh 2025-03-14T04:30:16.1902015Z remove-existing-keys: true 2025-03-14T04:30:16.1902322Z fail-silently: true 2025-03-14T04:30:16.1902607Z env: 2025-03-14T04:30:16.1902855Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:16.1903163Z ##[endgroup] 2025-03-14T04:30:16.3172119Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-03-14T04:30:16.3173840Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-03-14T04:30:16.3342899Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-03-14T04:30:16.3343527Z with: 2025-03-14T04:30:16.3343880Z no-sudo: true 2025-03-14T04:30:16.3344299Z submodules: recursive 2025-03-14T04:30:16.3344734Z fetch-depth: 0 2025-03-14T04:30:16.3345076Z env: 2025-03-14T04:30:16.3345477Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:16.3345873Z ##[endgroup] 2025-03-14T04:30:16.3437504Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T04:30:16.3438745Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T04:30:16.3446931Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:16.3447533Z env: 2025-03-14T04:30:16.3447920Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:16.3448359Z ##[endgroup] 2025-03-14T04:30:16.3544036Z ##[group]Run # Use all available CPUs for fetching 2025-03-14T04:30:16.3544518Z # Use all available CPUs for fetching 2025-03-14T04:30:16.3544934Z cd "${GITHUB_WORKSPACE}" 2025-03-14T04:30:16.3545345Z git config --global fetch.parallel 0 2025-03-14T04:30:16.3545784Z git config --global submodule.fetchJobs 0 2025-03-14T04:30:16.3546169Z  2025-03-14T04:30:16.3546644Z # Clean workspace. The default checkout action should also do this, but 2025-03-14T04:30:16.3547188Z # do it here as well just in case 2025-03-14T04:30:16.3547555Z if [[ -d .git ]]; then 2025-03-14T04:30:16.3547890Z  if [ -z "${NO_SUDO}" ]; then 2025-03-14T04:30:16.3548246Z  sudo git clean -ffdx 2025-03-14T04:30:16.3548557Z  else 2025-03-14T04:30:16.3548829Z  git clean -ffdx 2025-03-14T04:30:16.3549139Z  fi 2025-03-14T04:30:16.3549390Z fi 2025-03-14T04:30:16.3555201Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:16.3555683Z env: 2025-03-14T04:30:16.3555968Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:16.3556301Z NO_SUDO: true 2025-03-14T04:30:16.3556590Z ##[endgroup] 2025-03-14T04:30:16.5065881Z Removing .additional_ci_files/ 2025-03-14T04:30:16.5066511Z Removing .pytest_cache/ 2025-03-14T04:30:16.5066841Z Removing build/ 2025-03-14T04:30:16.5067135Z Removing dist/ 2025-03-14T04:30:16.5067675Z Removing logs-test-distributed-3-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38749776250.zip 2025-03-14T04:30:16.5068578Z Removing test-jsons-test-distributed-3-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38749776250.zip 2025-03-14T04:30:16.5069712Z Removing test-reports-test-distributed-3-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38749776250.zip 2025-03-14T04:30:16.5070432Z Removing test/.pytorch-disabled-tests.json 2025-03-14T04:30:16.5070847Z Removing test/__pycache__/ 2025-03-14T04:30:16.5071449Z Removing test/distributed/__pycache__/ 2025-03-14T04:30:16.5071977Z Removing test/test-reports/ 2025-03-14T04:30:16.5072352Z Removing tools/__pycache__/ 2025-03-14T04:30:16.5072802Z Removing tools/stats/__pycache__/ 2025-03-14T04:30:16.5073360Z Removing tools/stats/upload_utilization_stats/__pycache__/ 2025-03-14T04:30:16.5073859Z Removing tools/testing/__pycache__/ 2025-03-14T04:30:16.5074516Z Removing tools/testing/target_determination/__pycache__/ 2025-03-14T04:30:16.5075148Z Removing tools/testing/target_determination/heuristics/__pycache__/ 2025-03-14T04:30:16.5075690Z Removing usage_log.txt 2025-03-14T04:30:16.5164601Z ##[group]Run actions/checkout@v4 2025-03-14T04:30:16.5164982Z with: 2025-03-14T04:30:16.5165409Z ref: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:30:16.5165927Z fetch-depth: 0 2025-03-14T04:30:16.5166205Z submodules: recursive 2025-03-14T04:30:16.5166504Z show-progress: false 2025-03-14T04:30:16.5166810Z repository: pytorch/pytorch 2025-03-14T04:30:16.5167273Z token: *** 2025-03-14T04:30:16.5167534Z ssh-strict: true 2025-03-14T04:30:16.5167806Z ssh-user: git 2025-03-14T04:30:16.5168276Z persist-credentials: true 2025-03-14T04:30:16.5168593Z clean: true 2025-03-14T04:30:16.5168879Z sparse-checkout-cone-mode: true 2025-03-14T04:30:16.5169203Z fetch-tags: false 2025-03-14T04:30:16.5169823Z lfs: false 2025-03-14T04:30:16.5170115Z set-safe-directory: true 2025-03-14T04:30:16.5170441Z env: 2025-03-14T04:30:16.5170741Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:16.5171063Z ##[endgroup] 2025-03-14T04:30:16.6353500Z Syncing repository: pytorch/pytorch 2025-03-14T04:30:16.6355091Z ##[group]Getting Git version info 2025-03-14T04:30:16.6355688Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-03-14T04:30:16.6356518Z [command]/usr/bin/git version 2025-03-14T04:30:16.6356879Z git version 2.47.1 2025-03-14T04:30:16.6368890Z ##[endgroup] 2025-03-14T04:30:16.6378781Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/7acc71f3-c38e-4680-912e-3da3d5d30cdc/.gitconfig' 2025-03-14T04:30:16.6392948Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7acc71f3-c38e-4680-912e-3da3d5d30cdc' before making global git config changes 2025-03-14T04:30:16.6394268Z Adding repository directory to the temporary git global config as a safe directory 2025-03-14T04:30:16.6397327Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-03-14T04:30:16.6435104Z [command]/usr/bin/git config --local --get remote.origin.url 2025-03-14T04:30:16.6454846Z https://github.com/pytorch/pytorch 2025-03-14T04:30:16.6470016Z ##[group]Removing previously created refs, to avoid conflicts 2025-03-14T04:30:16.6473642Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD 2025-03-14T04:30:16.6494487Z HEAD 2025-03-14T04:30:16.6531722Z ##[endgroup] 2025-03-14T04:30:16.6533451Z [command]/usr/bin/git submodule status 2025-03-14T04:30:16.6891545Z 7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe) 2025-03-14T04:30:16.6971534Z 4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081) 2025-03-14T04:30:16.7054051Z b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327) 2025-03-14T04:30:16.7150534Z c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0) 2025-03-14T04:30:16.7174282Z e170594ac7cf1dac584da473d4ca9301087090c1 third_party/NVTX (v3.1.0) 2025-03-14T04:30:16.7251665Z a6bfc237255a6bac1513f7c1ebde6d8aed6b5191 third_party/VulkanMemoryAllocator (v2.1.0-705-ga6bfc23) 2025-03-14T04:30:16.7774921Z 51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a010365) 2025-03-14T04:30:16.7799675Z 0d98dba29d66e93259db7daa53a9327df767a415 third_party/benchmark (v1.6.1) 2025-03-14T04:30:16.7832078Z 8086bbe3a78d931eb96fe12fdc014082e18d18d3 third_party/composable_kernel (mock-tag-test-6-g8086bbe3a) 2025-03-14T04:30:16.7971363Z 3b6597bba913d51161383657829b7e644e59c006 third_party/cpp-httplib (v0.15.3-20-g3b6597b) 2025-03-14T04:30:16.8090758Z 1e83a2fdd3102f65c6f1fb602c1b320486218a99 third_party/cpuinfo (1e83a2f) 2025-03-14T04:30:16.8129325Z 91b7532f3386768bba4f444ee7672b497f34da8a third_party/cudnn_frontend (v0.5-44-g91b7532) 2025-03-14T04:30:16.8226110Z afa1772203677c5118fcd82537a9c8fefbcc7008 third_party/cutlass (v3.8.0) 2025-03-14T04:30:16.8898613Z 3147391d946bb4b6c68edd901f2add6ac1f31f8c third_party/eigen (3.4.0) 2025-03-14T04:30:16.9268521Z dbc3157bf256f1339b3fa1fef2be89ac4078be0e third_party/fbgemm (v0.4.1-446-gdbc3157b) 2025-03-14T04:30:16.9359156Z 979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4) 2025-03-14T04:30:16.9385854Z 01834de25e4bf3975a9a00e816292b1ad0fe184b third_party/flatbuffers (v23.3.3) 2025-03-14T04:30:16.9821676Z 123913715afeb8a437e6388b4473fcc4753e1c9a third_party/fmt (11.1.4) 2025-03-14T04:30:16.9931165Z 3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17) 2025-03-14T04:30:17.0052686Z 5354032ea08eadd7fc4456477f7f7c6308818509 third_party/gloo (5354032) 2025-03-14T04:30:17.0306835Z b514bdc898e2951020cbdca1304b75f5950d1f59 third_party/googletest (release-1.8.0-3484-gb514bdc8) 2025-03-14T04:30:17.0394149Z 719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1) 2025-03-14T04:30:17.0458616Z 5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42 third_party/ittapi (v3.23.0-14-g5b8a7d7) 2025-03-14T04:30:17.0731936Z 2859721fd9e73d3ca1c56f827dbc64e6d68f78a2 third_party/kineto (remotes/origin/sraikund/test-53-g2859721) 2025-03-14T04:30:17.0757518Z ef685a13cfbe8d418aa2ed34350e21e4938358b6 third_party/kleidiai (v1.3.0) 2025-03-14T04:30:17.0784237Z b66e3214d8a104669c2ec05ae91ebc26a8f5ab78 third_party/mimalloc (v1.8.2) 2025-03-14T04:30:17.1272777Z 87cda1d6646592ac5866dc703c8e1839046a6806 third_party/nlohmann (v3.10.1-113-g87cda1d6) 2025-03-14T04:30:17.1579905Z b8baa8446686496da4cc8fda09f2b6fe65c2a02c third_party/onnx (v1.17.0) 2025-03-14T04:30:17.1604769Z a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2) 2025-03-14T04:30:17.1632694Z 9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe third_party/pocketfft (release_for_eigen-11-g9d3ab05) 2025-03-14T04:30:17.2058790Z d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b) 2025-03-14T04:30:17.2136221Z 072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master) 2025-03-14T04:30:17.2195914Z 4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e) 2025-03-14T04:30:17.2249750Z a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e) 2025-03-14T04:30:17.2331366Z f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated) 2025-03-14T04:30:17.2449692Z 56e1f79cb140fb9326d612d0be06b5250565cade third_party/sleef (3.7-33-g56e1f79) 2025-03-14T04:30:17.2520385Z 52791a2fd214b2a9dc5759d36725909c1daa7f2e third_party/tensorpipe (remotes/origin/master) 2025-03-14T04:30:17.2532041Z ##[group]Cleaning the repository 2025-03-14T04:30:17.2537428Z [command]/usr/bin/git clean -ffdx 2025-03-14T04:30:17.2745850Z [command]/usr/bin/git reset --hard HEAD 2025-03-14T04:30:18.3271327Z Updating files: 79% (13135/16540) 2025-03-14T04:30:18.3482340Z Updating files: 80% (13232/16540) 2025-03-14T04:30:18.3604839Z Updating files: 81% (13398/16540) 2025-03-14T04:30:18.3735933Z Updating files: 82% (13563/16540) 2025-03-14T04:30:18.3884153Z Updating files: 83% (13729/16540) 2025-03-14T04:30:18.4033963Z Updating files: 84% (13894/16540) 2025-03-14T04:30:18.4153872Z Updating files: 85% (14059/16540) 2025-03-14T04:30:18.4283250Z Updating files: 86% (14225/16540) 2025-03-14T04:30:18.4399597Z Updating files: 87% (14390/16540) 2025-03-14T04:30:18.4545702Z Updating files: 88% (14556/16540) 2025-03-14T04:30:18.4724091Z Updating files: 89% (14721/16540) 2025-03-14T04:30:18.4840661Z Updating files: 90% (14886/16540) 2025-03-14T04:30:18.4971177Z Updating files: 91% (15052/16540) 2025-03-14T04:30:18.5128072Z Updating files: 92% (15217/16540) 2025-03-14T04:30:18.5314414Z Updating files: 93% (15383/16540) 2025-03-14T04:30:18.5509766Z Updating files: 94% (15548/16540) 2025-03-14T04:30:18.5668376Z Updating files: 95% (15713/16540) 2025-03-14T04:30:18.5811679Z Updating files: 96% (15879/16540) 2025-03-14T04:30:18.6101669Z Updating files: 97% (16044/16540) 2025-03-14T04:30:18.6262307Z Updating files: 98% (16210/16540) 2025-03-14T04:30:18.6423228Z Updating files: 99% (16375/16540) 2025-03-14T04:30:18.6423826Z Updating files: 100% (16540/16540) 2025-03-14T04:30:18.6424215Z Updating files: 100% (16540/16540), done. 2025-03-14T04:30:18.6599446Z HEAD is now at 15cd6921a56 [export] Fix tensor_constant and buffer naming conflicts in TS converter (#148803) 2025-03-14T04:30:18.6669792Z ##[endgroup] 2025-03-14T04:30:18.6670328Z ##[group]Disabling automatic garbage collection 2025-03-14T04:30:18.6676465Z [command]/usr/bin/git config --local gc.auto 0 2025-03-14T04:30:18.6706882Z ##[endgroup] 2025-03-14T04:30:18.6707387Z ##[group]Setting up auth 2025-03-14T04:30:18.6712917Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-03-14T04:30:18.6741752Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-03-14T04:30:18.7058502Z Entering 'android/libs/fbjni' 2025-03-14T04:30:18.7115652Z Entering 'third_party/FP16' 2025-03-14T04:30:18.7172815Z Entering 'third_party/FXdiv' 2025-03-14T04:30:18.7229748Z Entering 'third_party/NNPACK' 2025-03-14T04:30:18.7288902Z Entering 'third_party/NVTX' 2025-03-14T04:30:18.7348549Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T04:30:18.7407920Z Entering 'third_party/XNNPACK' 2025-03-14T04:30:18.7480205Z Entering 'third_party/benchmark' 2025-03-14T04:30:18.7537640Z Entering 'third_party/composable_kernel' 2025-03-14T04:30:18.7611722Z Entering 'third_party/cpp-httplib' 2025-03-14T04:30:18.7670009Z Entering 'third_party/cpuinfo' 2025-03-14T04:30:18.7727040Z Entering 'third_party/cudnn_frontend' 2025-03-14T04:30:18.7790929Z Entering 'third_party/cutlass' 2025-03-14T04:30:18.7857552Z Entering 'third_party/eigen' 2025-03-14T04:30:18.7915114Z Entering 'third_party/fbgemm' 2025-03-14T04:30:18.7971236Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T04:30:18.8027754Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T04:30:18.8082121Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T04:30:18.8145090Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T04:30:18.8205525Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T04:30:18.8261519Z Entering 'third_party/flash-attention' 2025-03-14T04:30:18.8323125Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T04:30:18.8385367Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T04:30:18.8456370Z Entering 'third_party/flatbuffers' 2025-03-14T04:30:18.8516477Z Entering 'third_party/fmt' 2025-03-14T04:30:18.8572122Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T04:30:18.8633838Z Entering 'third_party/gloo' 2025-03-14T04:30:18.8692910Z Entering 'third_party/googletest' 2025-03-14T04:30:18.8750499Z Entering 'third_party/ideep' 2025-03-14T04:30:18.8809616Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T04:30:18.8873637Z Entering 'third_party/ittapi' 2025-03-14T04:30:18.8930351Z Entering 'third_party/kineto' 2025-03-14T04:30:18.8987580Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T04:30:18.9042088Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T04:30:18.9100401Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T04:30:18.9158853Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T04:30:18.9219441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T04:30:18.9272986Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T04:30:18.9333699Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T04:30:18.9389471Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T04:30:18.9451768Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T04:30:18.9512914Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T04:30:18.9570829Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T04:30:18.9627376Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T04:30:18.9694332Z Entering 'third_party/kleidiai' 2025-03-14T04:30:18.9750247Z Entering 'third_party/mimalloc' 2025-03-14T04:30:18.9809968Z Entering 'third_party/nlohmann' 2025-03-14T04:30:18.9869567Z Entering 'third_party/onnx' 2025-03-14T04:30:18.9946489Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T04:30:19.0004885Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T04:30:19.0064331Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T04:30:19.0130604Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T04:30:19.0189981Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T04:30:19.0249798Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T04:30:19.0307396Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T04:30:19.0370597Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T04:30:19.0429616Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T04:30:19.0485274Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T04:30:19.0543842Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T04:30:19.0601883Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T04:30:19.0680518Z Entering 'third_party/pocketfft' 2025-03-14T04:30:19.0735852Z Entering 'third_party/protobuf' 2025-03-14T04:30:19.0795143Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T04:30:19.0851077Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T04:30:19.0908666Z Entering 'third_party/psimd' 2025-03-14T04:30:19.0965582Z Entering 'third_party/pthreadpool' 2025-03-14T04:30:19.1032132Z Entering 'third_party/pybind11' 2025-03-14T04:30:19.1089552Z Entering 'third_party/python-peachpy' 2025-03-14T04:30:19.1146104Z Entering 'third_party/sleef' 2025-03-14T04:30:19.1203877Z Entering 'third_party/tensorpipe' 2025-03-14T04:30:19.1260138Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T04:30:19.1315152Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T04:30:19.1371370Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T04:30:19.1428098Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T04:30:19.1481140Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T04:30:19.1555376Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-03-14T04:30:19.1584782Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-03-14T04:30:19.1898172Z Entering 'android/libs/fbjni' 2025-03-14T04:30:19.1955041Z Entering 'third_party/FP16' 2025-03-14T04:30:19.2013148Z Entering 'third_party/FXdiv' 2025-03-14T04:30:19.2070291Z Entering 'third_party/NNPACK' 2025-03-14T04:30:19.2125178Z Entering 'third_party/NVTX' 2025-03-14T04:30:19.2187307Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T04:30:19.2244842Z Entering 'third_party/XNNPACK' 2025-03-14T04:30:19.2328840Z Entering 'third_party/benchmark' 2025-03-14T04:30:19.2387924Z Entering 'third_party/composable_kernel' 2025-03-14T04:30:19.2459363Z Entering 'third_party/cpp-httplib' 2025-03-14T04:30:19.2518086Z Entering 'third_party/cpuinfo' 2025-03-14T04:30:19.2573804Z Entering 'third_party/cudnn_frontend' 2025-03-14T04:30:19.2631897Z Entering 'third_party/cutlass' 2025-03-14T04:30:19.2698246Z Entering 'third_party/eigen' 2025-03-14T04:30:19.2755045Z Entering 'third_party/fbgemm' 2025-03-14T04:30:19.2814060Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T04:30:19.2869680Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T04:30:19.2924925Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T04:30:19.2995158Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T04:30:19.3051101Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T04:30:19.3109851Z Entering 'third_party/flash-attention' 2025-03-14T04:30:19.3165149Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T04:30:19.3232382Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T04:30:19.3299499Z Entering 'third_party/flatbuffers' 2025-03-14T04:30:19.3357676Z Entering 'third_party/fmt' 2025-03-14T04:30:19.3415432Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T04:30:19.3471901Z Entering 'third_party/gloo' 2025-03-14T04:30:19.3530010Z Entering 'third_party/googletest' 2025-03-14T04:30:19.3588093Z Entering 'third_party/ideep' 2025-03-14T04:30:19.3642846Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T04:30:19.3713210Z Entering 'third_party/ittapi' 2025-03-14T04:30:19.3771372Z Entering 'third_party/kineto' 2025-03-14T04:30:19.3828347Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T04:30:19.3882367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T04:30:19.3943613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T04:30:19.4006671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T04:30:19.4071200Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T04:30:19.4130253Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T04:30:19.4190342Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T04:30:19.4251641Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T04:30:19.4310307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T04:30:19.4373685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T04:30:19.4433475Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T04:30:19.4490484Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T04:30:19.4548171Z Entering 'third_party/kleidiai' 2025-03-14T04:30:19.4605172Z Entering 'third_party/mimalloc' 2025-03-14T04:30:19.4663863Z Entering 'third_party/nlohmann' 2025-03-14T04:30:19.4736221Z Entering 'third_party/onnx' 2025-03-14T04:30:19.4811576Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T04:30:19.4871108Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T04:30:19.4932670Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T04:30:19.4988904Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T04:30:19.5053149Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T04:30:19.5110728Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T04:30:19.5171685Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T04:30:19.5228042Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T04:30:19.5292105Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T04:30:19.5348351Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T04:30:19.5410595Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T04:30:19.5471127Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T04:30:19.5553902Z Entering 'third_party/pocketfft' 2025-03-14T04:30:19.5614427Z Entering 'third_party/protobuf' 2025-03-14T04:30:19.5675292Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T04:30:19.5740681Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T04:30:19.5796467Z Entering 'third_party/psimd' 2025-03-14T04:30:19.5852814Z Entering 'third_party/pthreadpool' 2025-03-14T04:30:19.5911111Z Entering 'third_party/pybind11' 2025-03-14T04:30:19.5969726Z Entering 'third_party/python-peachpy' 2025-03-14T04:30:19.6032165Z Entering 'third_party/sleef' 2025-03-14T04:30:19.6089978Z Entering 'third_party/tensorpipe' 2025-03-14T04:30:19.6148185Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T04:30:19.6203154Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T04:30:19.6259656Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T04:30:19.6318306Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T04:30:19.6371733Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T04:30:19.6451217Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-03-14T04:30:19.6503111Z ##[endgroup] 2025-03-14T04:30:19.6503733Z ##[group]Fetching the repository 2025-03-14T04:30:19.6509598Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-03-14T04:30:19.9282495Z From https://github.com/pytorch/pytorch 2025-03-14T04:30:19.9283097Z - [deleted] (none) -> origin/angelayi/hf_version_update 2025-03-14T04:30:19.9710873Z - [deleted] (none) -> origin/atalman-patch-4 2025-03-14T04:30:19.9711567Z - [deleted] (none) -> origin/chenyang78/dyn-shape-ci-tmp 2025-03-14T04:30:19.9712165Z - [deleted] (none) -> origin/cleanup_vs_2019 2025-03-14T04:30:19.9712761Z - [deleted] (none) -> origin/csl/fflint 2025-03-14T04:30:19.9713579Z - [deleted] (none) -> origin/dataclass 2025-03-14T04:30:19.9714670Z - [deleted] (none) -> origin/gh/EikanWang/74/base 2025-03-14T04:30:19.9715866Z - [deleted] (none) -> origin/gh/EikanWang/74/head 2025-03-14T04:30:19.9717083Z - [deleted] (none) -> origin/gh/EikanWang/74/orig 2025-03-14T04:30:19.9718297Z - [deleted] (none) -> origin/gh/SamGinzburg/14/base 2025-03-14T04:30:19.9719523Z - [deleted] (none) -> origin/gh/SamGinzburg/14/head 2025-03-14T04:30:19.9720754Z - [deleted] (none) -> origin/gh/SamGinzburg/14/orig 2025-03-14T04:30:19.9722148Z - [deleted] (none) -> origin/gh/ZhiweiYan-96/37/base 2025-03-14T04:30:19.9723242Z - [deleted] (none) -> origin/gh/ZhiweiYan-96/37/head 2025-03-14T04:30:19.9724459Z - [deleted] (none) -> origin/gh/anijain2305/678/base 2025-03-14T04:30:19.9726188Z - [deleted] (none) -> origin/gh/anijain2305/678/head 2025-03-14T04:30:19.9727019Z - [deleted] (none) -> origin/gh/anijain2305/678/orig 2025-03-14T04:30:19.9728203Z - [deleted] (none) -> origin/gh/benjaminglass1/47/base 2025-03-14T04:30:19.9729395Z - [deleted] (none) -> origin/gh/benjaminglass1/47/head 2025-03-14T04:30:19.9730597Z - [deleted] (none) -> origin/gh/benjaminglass1/47/orig 2025-03-14T04:30:19.9731780Z - [deleted] (none) -> origin/gh/bobrenjc93/271/base 2025-03-14T04:30:19.9732988Z - [deleted] (none) -> origin/gh/bobrenjc93/271/head 2025-03-14T04:30:19.9734174Z - [deleted] (none) -> origin/gh/bobrenjc93/271/orig 2025-03-14T04:30:19.9735353Z - [deleted] (none) -> origin/gh/drisspg/124/base 2025-03-14T04:30:19.9736652Z - [deleted] (none) -> origin/gh/drisspg/124/head 2025-03-14T04:30:19.9737738Z - [deleted] (none) -> origin/gh/drisspg/124/orig 2025-03-14T04:30:19.9738970Z - [deleted] (none) -> origin/gh/int3/10/base 2025-03-14T04:30:19.9740121Z - [deleted] (none) -> origin/gh/int3/10/head 2025-03-14T04:30:19.9741363Z - [deleted] (none) -> origin/gh/int3/10/orig 2025-03-14T04:30:19.9742514Z - [deleted] (none) -> origin/gh/int3/11/base 2025-03-14T04:30:19.9743793Z - [deleted] (none) -> origin/gh/int3/11/head 2025-03-14T04:30:19.9745023Z - [deleted] (none) -> origin/gh/int3/11/orig 2025-03-14T04:30:19.9746092Z - [deleted] (none) -> origin/gh/int3/9/base 2025-03-14T04:30:19.9747337Z - [deleted] (none) -> origin/gh/int3/9/head 2025-03-14T04:30:19.9748481Z - [deleted] (none) -> origin/gh/int3/9/orig 2025-03-14T04:30:19.9749733Z - [deleted] (none) -> origin/gh/jamesjwu/106/base 2025-03-14T04:30:19.9750873Z - [deleted] (none) -> origin/gh/jamesjwu/106/head 2025-03-14T04:30:19.9752105Z - [deleted] (none) -> origin/gh/jamesjwu/106/orig 2025-03-14T04:30:19.9753523Z - [deleted] (none) -> origin/gh/kwen2501/124/base 2025-03-14T04:30:19.9754827Z - [deleted] (none) -> origin/gh/kwen2501/124/head 2025-03-14T04:30:19.9756002Z - [deleted] (none) -> origin/gh/kwen2501/124/orig 2025-03-14T04:30:19.9757317Z - [deleted] (none) -> origin/gh/laithsakka/102/base 2025-03-14T04:30:19.9758470Z - [deleted] (none) -> origin/gh/laithsakka/102/head 2025-03-14T04:30:19.9759763Z - [deleted] (none) -> origin/gh/laithsakka/102/orig 2025-03-14T04:30:19.9760933Z - [deleted] (none) -> origin/gh/masnesral/140/base 2025-03-14T04:30:19.9762150Z - [deleted] (none) -> origin/gh/masnesral/140/head 2025-03-14T04:30:19.9763438Z - [deleted] (none) -> origin/gh/masnesral/140/orig 2025-03-14T04:30:19.9764628Z - [deleted] (none) -> origin/gh/mikaylagawarecki/311/base 2025-03-14T04:30:19.9765944Z - [deleted] (none) -> origin/gh/mikaylagawarecki/311/head 2025-03-14T04:30:19.9767134Z - [deleted] (none) -> origin/gh/mikaylagawarecki/311/orig 2025-03-14T04:30:19.9768332Z - [deleted] (none) -> origin/gh/mikaylagawarecki/312/base 2025-03-14T04:30:19.9769696Z - [deleted] (none) -> origin/gh/mikaylagawarecki/312/head 2025-03-14T04:30:19.9771009Z - [deleted] (none) -> origin/gh/mikaylagawarecki/312/orig 2025-03-14T04:30:19.9772127Z - [deleted] (none) -> origin/gh/shunting314/198/base 2025-03-14T04:30:19.9773379Z - [deleted] (none) -> origin/gh/shunting314/198/head 2025-03-14T04:30:19.9774509Z - [deleted] (none) -> origin/gh/shunting314/198/orig 2025-03-14T04:30:19.9775721Z - [deleted] (none) -> origin/gh/tugsbayasgalan/288/base 2025-03-14T04:30:19.9776911Z - [deleted] (none) -> origin/gh/tugsbayasgalan/288/head 2025-03-14T04:30:19.9778092Z - [deleted] (none) -> origin/gh/tugsbayasgalan/288/orig 2025-03-14T04:30:19.9779406Z - [deleted] (none) -> origin/gh/voznesenskym/211/base 2025-03-14T04:30:19.9780481Z - [deleted] (none) -> origin/gh/voznesenskym/211/head 2025-03-14T04:30:19.9781743Z - [deleted] (none) -> origin/gh/voznesenskym/211/orig 2025-03-14T04:30:19.9782871Z - [deleted] (none) -> origin/gh/wconstab/382/base 2025-03-14T04:30:19.9784129Z - [deleted] (none) -> origin/gh/wconstab/383/base 2025-03-14T04:30:19.9785259Z - [deleted] (none) -> origin/gh/wconstab/384/base 2025-03-14T04:30:19.9786616Z - [deleted] (none) -> origin/gh/wconstab/385/base 2025-03-14T04:30:19.9787650Z - [deleted] (none) -> origin/gh/wconstab/385/head 2025-03-14T04:30:19.9788835Z - [deleted] (none) -> origin/gh/wconstab/386/base 2025-03-14T04:30:19.9790083Z - [deleted] (none) -> origin/gh/wconstab/391/base 2025-03-14T04:30:19.9791222Z - [deleted] (none) -> origin/gh/wconstab/391/head 2025-03-14T04:30:19.9792455Z - [deleted] (none) -> origin/gh/wconstab/391/orig 2025-03-14T04:30:19.9794029Z - [deleted] (none) -> origin/gh/yanboliang/63/base 2025-03-14T04:30:19.9795146Z - [deleted] (none) -> origin/gh/yanboliang/63/orig 2025-03-14T04:30:19.9796542Z - [deleted] (none) -> origin/gh/yanboliang/64/base 2025-03-14T04:30:19.9797607Z - [deleted] (none) -> origin/gh/yanboliang/64/orig 2025-03-14T04:30:19.9798819Z - [deleted] (none) -> origin/gh/yanboliang/65/base 2025-03-14T04:30:19.9800220Z - [deleted] (none) -> origin/gh/yanboliang/65/orig 2025-03-14T04:30:19.9801272Z - [deleted] (none) -> origin/gh/zou3519/1105/base 2025-03-14T04:30:19.9802575Z - [deleted] (none) -> origin/gh/zou3519/1105/head 2025-03-14T04:30:19.9803722Z - [deleted] (none) -> origin/gh/zou3519/1105/orig 2025-03-14T04:30:19.9805013Z - [deleted] (none) -> origin/gh/zou3519/1131/base 2025-03-14T04:30:19.9806270Z - [deleted] (none) -> origin/gh/zou3519/1131/head 2025-03-14T04:30:19.9807531Z - [deleted] (none) -> origin/gh/zou3519/1131/orig 2025-03-14T04:30:19.9808690Z - [deleted] (none) -> origin/gh/zou3519/1133/base 2025-03-14T04:30:19.9809853Z - [deleted] (none) -> origin/gh/zou3519/1133/head 2025-03-14T04:30:19.9811111Z - [deleted] (none) -> origin/gh/zou3519/1133/orig 2025-03-14T04:30:19.9812242Z - [deleted] (none) -> origin/malfet-patch-13 2025-03-14T04:30:19.9813500Z - [deleted] (none) -> origin/malfet-patch-14 2025-03-14T04:30:19.9814625Z - [deleted] (none) -> origin/malfet-patch-15 2025-03-14T04:30:19.9815873Z - [deleted] (none) -> origin/malfet-patch-4 2025-03-14T04:30:19.9817004Z - [deleted] (none) -> origin/malfet-patch-7 2025-03-14T04:30:19.9818199Z - [deleted] (none) -> origin/malfet-patch-9 2025-03-14T04:30:19.9819514Z - [deleted] (none) -> origin/malfet/enable-angle-atan2-for-int64 2025-03-14T04:30:19.9820631Z - [deleted] (none) -> origin/malfet/make-compile-shader-public 2025-03-14T04:30:19.9821792Z - [deleted] (none) -> origin/revert_commit_b57b38b 2025-03-14T04:30:19.9823063Z - [deleted] (none) -> origin/shunting-loaf-bm-6 2025-03-14T04:30:19.9824170Z - [deleted] (none) -> origin/zbv_algo 2025-03-14T04:30:19.9825436Z - [deleted] (none) -> ciflow/inductor/149162 2025-03-14T04:30:19.9826559Z - [deleted] (none) -> ciflow/inductor/149167 2025-03-14T04:30:19.9827812Z - [deleted] (none) -> ciflow/inductor/149174 2025-03-14T04:30:19.9829020Z - [deleted] (none) -> ciflow/mps/149174 2025-03-14T04:30:19.9830145Z - [deleted] (none) -> ciflow/trunk/134515 2025-03-14T04:30:19.9831463Z - [deleted] (none) -> ciflow/trunk/145197 2025-03-14T04:30:19.9832531Z - [deleted] (none) -> ciflow/trunk/148677 2025-03-14T04:30:19.9834070Z - [deleted] (none) -> ciflow/trunk/149142 2025-03-14T04:30:19.9835237Z - [deleted] (none) -> ciflow/trunk/149152 2025-03-14T04:30:19.9836520Z - [deleted] (none) -> ciflow/trunk/149166 2025-03-14T04:30:21.1562575Z * [new branch] gh/etaf/110/base -> origin/gh/etaf/110/base 2025-03-14T04:30:21.1563705Z * [new branch] gh/etaf/110/head -> origin/gh/etaf/110/head 2025-03-14T04:30:21.1564584Z * [new branch] gh/etaf/110/orig -> origin/gh/etaf/110/orig 2025-03-14T04:30:21.1580833Z 8636b394e08..9db33f3cda6 gh/malfet/227/base -> origin/gh/malfet/227/base 2025-03-14T04:30:21.1581626Z 259919aeab3..4b1f10d8c28 gh/malfet/227/head -> origin/gh/malfet/227/head 2025-03-14T04:30:21.1582845Z + 4c129ff8a05...13ad80715a8 gh/malfet/227/orig -> origin/gh/malfet/227/orig (forced update) 2025-03-14T04:30:21.1598510Z e1c0f487ca2..bd72c961cb2 jnair/mi300_docker_caching_workflow -> origin/jnair/mi300_docker_caching_workflow 2025-03-14T04:30:21.1600584Z 15cd6921a56..bea181ff7ee main -> origin/main 2025-03-14T04:30:21.1607004Z + 277cb23fd22...c4b8be3d09f update_submodule_FBGEMM -> origin/update_submodule_FBGEMM (forced update) 2025-03-14T04:30:21.1607929Z e5fccb2babe..49570cb4024 viable/strict -> origin/viable/strict 2025-03-14T04:30:21.1609372Z 1579a02c823..3ffd7552df1 wdvr/iss145259_alt -> origin/wdvr/iss145259_alt 2025-03-14T04:30:21.1616263Z t [tag update] ciflow/inductor/148424 -> ciflow/inductor/148424 2025-03-14T04:30:21.1618115Z t [tag update] ciflow/inductor/149027 -> ciflow/inductor/149027 2025-03-14T04:30:21.1619753Z * [new tag] ciflow/inductor/149173 -> ciflow/inductor/149173 2025-03-14T04:30:21.1620399Z * [new tag] ciflow/inductor/149176 -> ciflow/inductor/149176 2025-03-14T04:30:21.1621537Z t [tag update] ciflow/linux-aarch64/148424 -> ciflow/linux-aarch64/148424 2025-03-14T04:30:21.1622650Z t [tag update] ciflow/mps/149123 -> ciflow/mps/149123 2025-03-14T04:30:21.1623344Z * [new tag] ciflow/mps/149173 -> ciflow/mps/149173 2025-03-14T04:30:21.1624532Z t [tag update] ciflow/rocm-mi300/148394 -> ciflow/rocm-mi300/148394 2025-03-14T04:30:21.1625627Z t [tag update] ciflow/rocm/148394 -> ciflow/rocm/148394 2025-03-14T04:30:21.1628819Z * [new tag] ciflow/trunk/146289 -> ciflow/trunk/146289 2025-03-14T04:30:21.1630751Z * [new tag] ciflow/trunk/149018 -> ciflow/trunk/149018 2025-03-14T04:30:21.1631431Z * [new tag] ciflow/trunk/149064 -> ciflow/trunk/149064 2025-03-14T04:30:21.1632700Z * [new tag] ciflow/xpu/149175 -> ciflow/xpu/149175 2025-03-14T04:30:21.2187661Z [command]/usr/bin/git rev-parse --verify --quiet aed0b7a742a2d7b7901790622829cbd2135049a4^{object} 2025-03-14T04:30:21.2213798Z aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:30:21.2216037Z ##[endgroup] 2025-03-14T04:30:21.2216569Z ##[group]Determining the checkout info 2025-03-14T04:30:21.2217403Z ##[endgroup] 2025-03-14T04:30:21.2222370Z [command]/usr/bin/git sparse-checkout disable 2025-03-14T04:30:21.5538242Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-03-14T04:30:21.5567909Z ##[group]Checking out the ref 2025-03-14T04:30:21.5571586Z [command]/usr/bin/git checkout --progress --force aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:30:21.6856209Z Previous HEAD position was 15cd6921a56 [export] Fix tensor_constant and buffer naming conflicts in TS converter (#148803) 2025-03-14T04:30:21.6862749Z HEAD is now at aed0b7a742a [c10d] Add param recording for uniqueID broadcasting and allgather (#149166) 2025-03-14T04:30:21.6892278Z ##[endgroup] 2025-03-14T04:30:21.6892832Z ##[group]Setting up auth for fetching submodules 2025-03-14T04:30:21.6897553Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-03-14T04:30:21.6943108Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-03-14T04:30:21.6972258Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-03-14T04:30:21.6998345Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-03-14T04:30:21.7022780Z ##[endgroup] 2025-03-14T04:30:21.7023372Z ##[group]Fetching submodules 2025-03-14T04:30:21.7025624Z [command]/usr/bin/git submodule sync --recursive 2025-03-14T04:30:21.7345702Z Synchronizing submodule url for 'android/libs/fbjni' 2025-03-14T04:30:21.7367981Z Synchronizing submodule url for 'third_party/FP16' 2025-03-14T04:30:21.7392336Z Synchronizing submodule url for 'third_party/FXdiv' 2025-03-14T04:30:21.7417635Z Synchronizing submodule url for 'third_party/NNPACK' 2025-03-14T04:30:21.7439123Z Synchronizing submodule url for 'third_party/NVTX' 2025-03-14T04:30:21.7463587Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator' 2025-03-14T04:30:21.7488281Z Synchronizing submodule url for 'third_party/XNNPACK' 2025-03-14T04:30:21.7524484Z Synchronizing submodule url for 'third_party/benchmark' 2025-03-14T04:30:21.7548261Z Synchronizing submodule url for 'third_party/composable_kernel' 2025-03-14T04:30:21.7577445Z Synchronizing submodule url for 'third_party/cpp-httplib' 2025-03-14T04:30:21.7599531Z Synchronizing submodule url for 'third_party/cpuinfo' 2025-03-14T04:30:21.7623695Z Synchronizing submodule url for 'third_party/cudnn_frontend' 2025-03-14T04:30:21.7648521Z Synchronizing submodule url for 'third_party/cutlass' 2025-03-14T04:30:21.7678688Z Synchronizing submodule url for 'third_party/eigen' 2025-03-14T04:30:21.7706062Z Synchronizing submodule url for 'third_party/fbgemm' 2025-03-14T04:30:21.7729105Z Synchronizing submodule url for 'third_party/fbgemm/third_party/asmjit' 2025-03-14T04:30:21.7749666Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T04:30:21.7771431Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cutlass' 2025-03-14T04:30:21.7799204Z Synchronizing submodule url for 'third_party/fbgemm/third_party/googletest' 2025-03-14T04:30:21.7822954Z Synchronizing submodule url for 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T04:30:21.7849268Z Synchronizing submodule url for 'third_party/flash-attention' 2025-03-14T04:30:21.7869954Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T04:30:21.7900229Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass' 2025-03-14T04:30:21.7938076Z Synchronizing submodule url for 'third_party/flatbuffers' 2025-03-14T04:30:21.7962477Z Synchronizing submodule url for 'third_party/fmt' 2025-03-14T04:30:21.7988076Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp' 2025-03-14T04:30:21.8011320Z Synchronizing submodule url for 'third_party/gloo' 2025-03-14T04:30:21.8033840Z Synchronizing submodule url for 'third_party/googletest' 2025-03-14T04:30:21.8057746Z Synchronizing submodule url for 'third_party/ideep' 2025-03-14T04:30:21.8076530Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn' 2025-03-14T04:30:21.8114115Z Synchronizing submodule url for 'third_party/ittapi' 2025-03-14T04:30:21.8137019Z Synchronizing submodule url for 'third_party/kineto' 2025-03-14T04:30:21.8158249Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T04:30:21.8181169Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T04:30:21.8204153Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T04:30:21.8228029Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T04:30:21.8251902Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T04:30:21.8271653Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T04:30:21.8301028Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T04:30:21.8320820Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T04:30:21.8345183Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T04:30:21.8368107Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T04:30:21.8393942Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T04:30:21.8416324Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T04:30:21.8438976Z Synchronizing submodule url for 'third_party/kleidiai' 2025-03-14T04:30:21.8463515Z Synchronizing submodule url for 'third_party/mimalloc' 2025-03-14T04:30:21.8488500Z Synchronizing submodule url for 'third_party/nlohmann' 2025-03-14T04:30:21.8512397Z Synchronizing submodule url for 'third_party/onnx' 2025-03-14T04:30:21.8550458Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11' 2025-03-14T04:30:21.8576624Z Synchronizing submodule url for 'third_party/opentelemetry-cpp' 2025-03-14T04:30:21.8599788Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T04:30:21.8622058Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T04:30:21.8640596Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T04:30:21.8663174Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T04:30:21.8687521Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T04:30:21.8708367Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T04:30:21.8730521Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T04:30:21.8750001Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T04:30:21.8772615Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T04:30:21.8797425Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T04:30:21.8844308Z Synchronizing submodule url for 'third_party/pocketfft' 2025-03-14T04:30:21.8867188Z Synchronizing submodule url for 'third_party/protobuf' 2025-03-14T04:30:21.8893120Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark' 2025-03-14T04:30:21.8913948Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest' 2025-03-14T04:30:21.8940300Z Synchronizing submodule url for 'third_party/psimd' 2025-03-14T04:30:21.8960528Z Synchronizing submodule url for 'third_party/pthreadpool' 2025-03-14T04:30:21.8986136Z Synchronizing submodule url for 'third_party/pybind11' 2025-03-14T04:30:21.9009996Z Synchronizing submodule url for 'third_party/python-peachpy' 2025-03-14T04:30:21.9035196Z Synchronizing submodule url for 'third_party/sleef' 2025-03-14T04:30:21.9059583Z Synchronizing submodule url for 'third_party/tensorpipe' 2025-03-14T04:30:21.9079691Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest' 2025-03-14T04:30:21.9102176Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop' 2025-03-14T04:30:21.9120076Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv' 2025-03-14T04:30:21.9144018Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T04:30:21.9161788Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T04:30:21.9198394Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-03-14T04:30:21.9695418Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-03-14T04:30:21.9828105Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-03-14T04:30:21.9925676Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-03-14T04:30:22.0220533Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-03-14T04:30:22.0649305Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2025-03-14T04:30:22.1099618Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2025-03-14T04:30:23.0357421Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-03-14T04:30:23.0630755Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2025-03-14T04:30:23.3757732Z Submodule path 'third_party/composable_kernel': checked out '8086bbe3a78d931eb96fe12fdc014082e18d18d3' 2025-03-14T04:30:23.4252224Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2025-03-14T04:30:23.5374814Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2025-03-14T04:30:23.5770275Z Submodule path 'third_party/cudnn_frontend': checked out '91b7532f3386768bba4f444ee7672b497f34da8a' 2025-03-14T04:30:24.3115393Z Submodule path 'third_party/cutlass': checked out 'afa1772203677c5118fcd82537a9c8fefbcc7008' 2025-03-14T04:30:24.6114769Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2025-03-14T04:30:24.6886217Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2025-03-14T04:30:24.7392949Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2025-03-14T04:30:24.8479259Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2025-03-14T04:30:25.3671614Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2025-03-14T04:30:25.4201034Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2025-03-14T04:30:25.4332114Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2025-03-14T04:30:25.5200321Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-03-14T04:30:25.8260176Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-03-14T04:30:26.5292123Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-03-14T04:30:26.7026993Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2025-03-14T04:30:26.7395017Z Submodule path 'third_party/fmt': checked out '123913715afeb8a437e6388b4473fcc4753e1c9a' 2025-03-14T04:30:26.7836404Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-03-14T04:30:26.8154187Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2025-03-14T04:30:26.8656522Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2025-03-14T04:30:26.8797910Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-03-14T04:30:27.5443459Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-03-14T04:30:27.5635995Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2025-03-14T04:30:27.6615863Z Submodule path 'third_party/kineto': checked out '2859721fd9e73d3ca1c56f827dbc64e6d68f78a2' 2025-03-14T04:30:27.7580072Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2025-03-14T04:30:27.9701459Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-03-14T04:30:27.9919705Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-03-14T04:30:28.0355513Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-03-14T04:30:28.0514717Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-03-14T04:30:28.0617901Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-03-14T04:30:28.0830712Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-03-14T04:30:28.1322221Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2025-03-14T04:30:28.2536269Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-03-14T04:30:28.2734357Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-03-14T04:30:28.3115238Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2025-03-14T04:30:28.3648999Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2025-03-14T04:30:28.4066462Z Submodule path 'third_party/kleidiai': checked out 'ef685a13cfbe8d418aa2ed34350e21e4938358b6' 2025-03-14T04:30:28.4498707Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2025-03-14T04:30:28.5741978Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2025-03-14T04:30:29.0298162Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2025-03-14T04:30:29.0741887Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2025-03-14T04:30:29.1652534Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-03-14T04:30:29.1898596Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-03-14T04:30:29.2355242Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-03-14T04:30:29.2498204Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-03-14T04:30:29.3747457Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-03-14T04:30:29.3884507Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-03-14T04:30:29.4060337Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-03-14T04:30:29.4257944Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-03-14T04:30:29.7176921Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-03-14T04:30:29.7715281Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-03-14T04:30:30.3866418Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-03-14T04:30:30.3990201Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2025-03-14T04:30:30.7237365Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-03-14T04:30:30.7419032Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-03-14T04:30:30.8001201Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-03-14T04:30:30.8101278Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-03-14T04:30:30.8235568Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-03-14T04:30:30.8663671Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-03-14T04:30:30.8995092Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-03-14T04:30:30.9474303Z Submodule path 'third_party/sleef': checked out '56e1f79cb140fb9326d612d0be06b5250565cade' 2025-03-14T04:30:30.9801402Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2025-03-14T04:30:31.0312505Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-03-14T04:30:31.0495077Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-03-14T04:30:31.1219993Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2025-03-14T04:30:31.1550334Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-03-14T04:30:31.1648931Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-03-14T04:30:31.1690357Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-03-14T04:30:31.2005089Z Entering 'android/libs/fbjni' 2025-03-14T04:30:31.2049842Z Entering 'third_party/FP16' 2025-03-14T04:30:31.2093512Z Entering 'third_party/FXdiv' 2025-03-14T04:30:31.2135654Z Entering 'third_party/NNPACK' 2025-03-14T04:30:31.2179557Z Entering 'third_party/NVTX' 2025-03-14T04:30:31.2223752Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T04:30:31.2269150Z Entering 'third_party/XNNPACK' 2025-03-14T04:30:31.2331875Z Entering 'third_party/benchmark' 2025-03-14T04:30:31.2374078Z Entering 'third_party/composable_kernel' 2025-03-14T04:30:31.2427979Z Entering 'third_party/cpp-httplib' 2025-03-14T04:30:31.2471072Z Entering 'third_party/cpuinfo' 2025-03-14T04:30:31.2514846Z Entering 'third_party/cudnn_frontend' 2025-03-14T04:30:31.2557393Z Entering 'third_party/cutlass' 2025-03-14T04:30:31.2614736Z Entering 'third_party/eigen' 2025-03-14T04:30:31.2663530Z Entering 'third_party/fbgemm' 2025-03-14T04:30:31.2713157Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T04:30:31.2754830Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T04:30:31.2796491Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T04:30:31.2848024Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T04:30:31.2890003Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T04:30:31.2932549Z Entering 'third_party/flash-attention' 2025-03-14T04:30:31.2975795Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T04:30:31.3026568Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T04:30:31.3080699Z Entering 'third_party/flatbuffers' 2025-03-14T04:30:31.3130612Z Entering 'third_party/fmt' 2025-03-14T04:30:31.3171924Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T04:30:31.3215864Z Entering 'third_party/gloo' 2025-03-14T04:30:31.3258572Z Entering 'third_party/googletest' 2025-03-14T04:30:31.3303149Z Entering 'third_party/ideep' 2025-03-14T04:30:31.3346524Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T04:30:31.3397799Z Entering 'third_party/ittapi' 2025-03-14T04:30:31.3439389Z Entering 'third_party/kineto' 2025-03-14T04:30:31.3483436Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T04:30:31.3531881Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T04:30:31.3572575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T04:30:31.3616151Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T04:30:31.3658916Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T04:30:31.3700323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T04:30:31.3747333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T04:30:31.3790134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T04:30:31.3834637Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T04:30:31.3877015Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T04:30:31.3932545Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T04:30:31.3972616Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T04:30:31.4016772Z Entering 'third_party/kleidiai' 2025-03-14T04:30:31.4059938Z Entering 'third_party/mimalloc' 2025-03-14T04:30:31.4105084Z Entering 'third_party/nlohmann' 2025-03-14T04:30:31.4150228Z Entering 'third_party/onnx' 2025-03-14T04:30:31.4212604Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T04:30:31.4257455Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T04:30:31.4302404Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T04:30:31.4347113Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T04:30:31.4390492Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T04:30:31.4434646Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T04:30:31.4478393Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T04:30:31.4520344Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T04:30:31.4568388Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T04:30:31.4610038Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T04:30:31.4654094Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T04:30:31.4701050Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T04:30:31.4771454Z Entering 'third_party/pocketfft' 2025-03-14T04:30:31.4815162Z Entering 'third_party/protobuf' 2025-03-14T04:30:31.4862517Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T04:30:31.4905905Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T04:30:31.4951222Z Entering 'third_party/psimd' 2025-03-14T04:30:31.4995658Z Entering 'third_party/pthreadpool' 2025-03-14T04:30:31.5038127Z Entering 'third_party/pybind11' 2025-03-14T04:30:31.5082450Z Entering 'third_party/python-peachpy' 2025-03-14T04:30:31.5128067Z Entering 'third_party/sleef' 2025-03-14T04:30:31.5171196Z Entering 'third_party/tensorpipe' 2025-03-14T04:30:31.5214654Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T04:30:31.5256569Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T04:30:31.5296736Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T04:30:31.5340166Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T04:30:31.5379816Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T04:30:31.5438436Z ##[endgroup] 2025-03-14T04:30:31.5438979Z ##[group]Persisting credentials for submodules 2025-03-14T04:30:31.5445673Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-03-14T04:30:31.5757184Z Entering 'android/libs/fbjni' 2025-03-14T04:30:31.5796453Z url.https://github.com/.insteadof 2025-03-14T04:30:31.5796912Z url.https://github.com/.insteadof 2025-03-14T04:30:31.5836039Z Entering 'third_party/FP16' 2025-03-14T04:30:31.5873409Z url.https://github.com/.insteadof 2025-03-14T04:30:31.5873889Z url.https://github.com/.insteadof 2025-03-14T04:30:31.5911483Z Entering 'third_party/FXdiv' 2025-03-14T04:30:31.5952792Z url.https://github.com/.insteadof 2025-03-14T04:30:31.5953352Z url.https://github.com/.insteadof 2025-03-14T04:30:31.5989480Z Entering 'third_party/NNPACK' 2025-03-14T04:30:31.6027019Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6027547Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6061820Z Entering 'third_party/NVTX' 2025-03-14T04:30:31.6100881Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6101774Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6136209Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T04:30:31.6173286Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6173760Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6211185Z Entering 'third_party/XNNPACK' 2025-03-14T04:30:31.6249272Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6249803Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6304426Z Entering 'third_party/benchmark' 2025-03-14T04:30:31.6342232Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6342768Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6376306Z Entering 'third_party/composable_kernel' 2025-03-14T04:30:31.6415700Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6416229Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6463195Z Entering 'third_party/cpp-httplib' 2025-03-14T04:30:31.6501473Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6502004Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6536394Z Entering 'third_party/cpuinfo' 2025-03-14T04:30:31.6573444Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6574160Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6613077Z Entering 'third_party/cudnn_frontend' 2025-03-14T04:30:31.6652305Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6653178Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6689157Z Entering 'third_party/cutlass' 2025-03-14T04:30:31.6727558Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6728036Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6781115Z Entering 'third_party/eigen' 2025-03-14T04:30:31.6821655Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6822114Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6859153Z Entering 'third_party/fbgemm' 2025-03-14T04:30:31.6898203Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6898753Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6933598Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T04:30:31.6970744Z url.https://github.com/.insteadof 2025-03-14T04:30:31.6971212Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7004244Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T04:30:31.7043139Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7043907Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7077252Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T04:30:31.7114363Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7114822Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7155302Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T04:30:31.7194375Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7194830Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7232758Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T04:30:31.7270919Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7271380Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7312645Z Entering 'third_party/flash-attention' 2025-03-14T04:30:31.7350230Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7350738Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7389729Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T04:30:31.7427065Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7427530Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7472139Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T04:30:31.7511959Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7512680Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7561554Z Entering 'third_party/flatbuffers' 2025-03-14T04:30:31.7600064Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7637787Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7638224Z Entering 'third_party/fmt' 2025-03-14T04:30:31.7675803Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7676267Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7711721Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T04:30:31.7750246Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7750765Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7791325Z Entering 'third_party/gloo' 2025-03-14T04:30:31.7831080Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7831600Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7870069Z Entering 'third_party/googletest' 2025-03-14T04:30:31.7907836Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7908665Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7943615Z Entering 'third_party/ideep' 2025-03-14T04:30:31.7984812Z url.https://github.com/.insteadof 2025-03-14T04:30:31.7985528Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8019170Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T04:30:31.8055719Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8056469Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8100487Z Entering 'third_party/ittapi' 2025-03-14T04:30:31.8138368Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8138880Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8171987Z Entering 'third_party/kineto' 2025-03-14T04:30:31.8210795Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8211468Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8254691Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T04:30:31.8293774Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8294546Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8333757Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T04:30:31.8370964Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8371425Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8414645Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T04:30:31.8452729Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8453194Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8489209Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T04:30:31.8528849Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8529438Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8569916Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T04:30:31.8607971Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8608716Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8643346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T04:30:31.8681156Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8681620Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8718906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T04:30:31.8757107Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8757575Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8794827Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T04:30:31.8832720Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8833756Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8869796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T04:30:31.8907188Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8907723Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8947558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T04:30:31.8987618Z url.https://github.com/.insteadof 2025-03-14T04:30:31.8988151Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9026183Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T04:30:31.9063594Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9064330Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9098153Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T04:30:31.9134442Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9135171Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9173784Z Entering 'third_party/kleidiai' 2025-03-14T04:30:31.9213387Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9214112Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9251769Z Entering 'third_party/mimalloc' 2025-03-14T04:30:31.9290868Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9291424Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9328516Z Entering 'third_party/nlohmann' 2025-03-14T04:30:31.9366346Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9366806Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9402911Z Entering 'third_party/onnx' 2025-03-14T04:30:31.9441802Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9442370Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9496784Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T04:30:31.9534097Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9534618Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9572039Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T04:30:31.9611333Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9612072Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9653150Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T04:30:31.9689644Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9690305Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9731761Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T04:30:31.9769423Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9770421Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9807123Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T04:30:31.9843787Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9844309Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9882785Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T04:30:31.9919788Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9958545Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9959151Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T04:30:31.9996891Z url.https://github.com/.insteadof 2025-03-14T04:30:31.9997358Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0035672Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T04:30:32.0072032Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0072483Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0108084Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T04:30:32.0144452Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0144971Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0178690Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T04:30:32.0216995Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0217740Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0255419Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T04:30:32.0293795Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0294227Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0334522Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T04:30:32.0371044Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0371499Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0434450Z Entering 'third_party/pocketfft' 2025-03-14T04:30:32.0471826Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0472277Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0508347Z Entering 'third_party/protobuf' 2025-03-14T04:30:32.0547161Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0547682Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0592567Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T04:30:32.0630636Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0631151Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0670684Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T04:30:32.0708128Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0708680Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0746989Z Entering 'third_party/psimd' 2025-03-14T04:30:32.0788141Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0788693Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0822968Z Entering 'third_party/pthreadpool' 2025-03-14T04:30:32.0862002Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0862746Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0897067Z Entering 'third_party/pybind11' 2025-03-14T04:30:32.0934760Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0935470Z url.https://github.com/.insteadof 2025-03-14T04:30:32.0971692Z Entering 'third_party/python-peachpy' 2025-03-14T04:30:32.1009738Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1010251Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1052362Z Entering 'third_party/sleef' 2025-03-14T04:30:32.1092212Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1093098Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1128848Z Entering 'third_party/tensorpipe' 2025-03-14T04:30:32.1169478Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1170114Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1205142Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T04:30:32.1242280Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1242744Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1289937Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T04:30:32.1325170Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1325817Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1362170Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T04:30:32.1400618Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1401080Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1439700Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T04:30:32.1476311Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1476775Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1518329Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T04:30:32.1555157Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1555607Z url.https://github.com/.insteadof 2025-03-14T04:30:32.1609957Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-03-14T04:30:32.1922951Z Entering 'android/libs/fbjni' 2025-03-14T04:30:32.1971815Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-03-14T04:30:32.1990385Z Entering 'third_party/FP16' 2025-03-14T04:30:32.2040555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-03-14T04:30:32.2058969Z Entering 'third_party/FXdiv' 2025-03-14T04:30:32.2111453Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-03-14T04:30:32.2129925Z Entering 'third_party/NNPACK' 2025-03-14T04:30:32.2182128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-03-14T04:30:32.2200748Z Entering 'third_party/NVTX' 2025-03-14T04:30:32.2253104Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-03-14T04:30:32.2270993Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T04:30:32.2321131Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-03-14T04:30:32.2339889Z Entering 'third_party/XNNPACK' 2025-03-14T04:30:32.2392791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-03-14T04:30:32.2427825Z Entering 'third_party/benchmark' 2025-03-14T04:30:32.2479134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-03-14T04:30:32.2497777Z Entering 'third_party/composable_kernel' 2025-03-14T04:30:32.2549105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-03-14T04:30:32.2573557Z Entering 'third_party/cpp-httplib' 2025-03-14T04:30:32.2627225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-03-14T04:30:32.2643133Z Entering 'third_party/cpuinfo' 2025-03-14T04:30:32.2697883Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-03-14T04:30:32.2714730Z Entering 'third_party/cudnn_frontend' 2025-03-14T04:30:32.2765730Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-03-14T04:30:32.2786930Z Entering 'third_party/cutlass' 2025-03-14T04:30:32.2846417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-03-14T04:30:32.2872905Z Entering 'third_party/eigen' 2025-03-14T04:30:32.2925111Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2025-03-14T04:30:32.2947187Z Entering 'third_party/fbgemm' 2025-03-14T04:30:32.3004146Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-03-14T04:30:32.3023368Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T04:30:32.3074017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2025-03-14T04:30:32.3091590Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T04:30:32.3143411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2025-03-14T04:30:32.3159192Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T04:30:32.3215419Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2025-03-14T04:30:32.3238675Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T04:30:32.3292217Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2025-03-14T04:30:32.3309072Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T04:30:32.3358937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2025-03-14T04:30:32.3376593Z Entering 'third_party/flash-attention' 2025-03-14T04:30:32.3429998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-03-14T04:30:32.3449068Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T04:30:32.3500696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-03-14T04:30:32.3522029Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T04:30:32.3573905Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-03-14T04:30:32.3601077Z Entering 'third_party/flatbuffers' 2025-03-14T04:30:32.3653723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-03-14T04:30:32.3674333Z Entering 'third_party/fmt' 2025-03-14T04:30:32.3725160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-03-14T04:30:32.3750659Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T04:30:32.3802860Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-03-14T04:30:32.3821555Z Entering 'third_party/gloo' 2025-03-14T04:30:32.3874136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-03-14T04:30:32.3892821Z Entering 'third_party/googletest' 2025-03-14T04:30:32.3944056Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-03-14T04:30:32.3960134Z Entering 'third_party/ideep' 2025-03-14T04:30:32.4016477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-03-14T04:30:32.4033435Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T04:30:32.4082709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-03-14T04:30:32.4110468Z Entering 'third_party/ittapi' 2025-03-14T04:30:32.4160176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-03-14T04:30:32.4177626Z Entering 'third_party/kineto' 2025-03-14T04:30:32.4230242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-03-14T04:30:32.4249140Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T04:30:32.4301517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-03-14T04:30:32.4316064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T04:30:32.4369504Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-03-14T04:30:32.4391059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T04:30:32.4440395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-03-14T04:30:32.4458700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T04:30:32.4510876Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-03-14T04:30:32.4530186Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T04:30:32.4585605Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-03-14T04:30:32.4599920Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T04:30:32.4652437Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-03-14T04:30:32.4671475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T04:30:32.4722450Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-03-14T04:30:32.4740925Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T04:30:32.4794638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-03-14T04:30:32.4813500Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T04:30:32.4864575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-03-14T04:30:32.4882094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T04:30:32.4935248Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-03-14T04:30:32.4954602Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T04:30:32.5009422Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-03-14T04:30:32.5026272Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T04:30:32.5077125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-03-14T04:30:32.5096115Z Entering 'third_party/kleidiai' 2025-03-14T04:30:32.5147630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-03-14T04:30:32.5163867Z Entering 'third_party/mimalloc' 2025-03-14T04:30:32.5217802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-03-14T04:30:32.5234609Z Entering 'third_party/nlohmann' 2025-03-14T04:30:32.5287363Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-03-14T04:30:32.5306995Z Entering 'third_party/onnx' 2025-03-14T04:30:32.5358289Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-03-14T04:30:32.5394356Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T04:30:32.5443937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-03-14T04:30:32.5465220Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T04:30:32.5518548Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-03-14T04:30:32.5538064Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T04:30:32.5591941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-03-14T04:30:32.5610046Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T04:30:32.5661707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-03-14T04:30:32.5677407Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T04:30:32.5730187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-03-14T04:30:32.5747596Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T04:30:32.5799976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-03-14T04:30:32.5823323Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T04:30:32.5879498Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-03-14T04:30:32.5897849Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T04:30:32.5948136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-03-14T04:30:32.5963407Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T04:30:32.6017839Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-03-14T04:30:32.6033932Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T04:30:32.6086438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-03-14T04:30:32.6106712Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T04:30:32.6162088Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-03-14T04:30:32.6179946Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T04:30:32.6236813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-03-14T04:30:32.6275154Z Entering 'third_party/pocketfft' 2025-03-14T04:30:32.6328105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-03-14T04:30:32.6345438Z Entering 'third_party/protobuf' 2025-03-14T04:30:32.6398536Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-03-14T04:30:32.6419953Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T04:30:32.6472093Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-03-14T04:30:32.6489832Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T04:30:32.6540939Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-03-14T04:30:32.6558355Z Entering 'third_party/psimd' 2025-03-14T04:30:32.6612417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-03-14T04:30:32.6629671Z Entering 'third_party/pthreadpool' 2025-03-14T04:30:32.6682626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-03-14T04:30:32.6700621Z Entering 'third_party/pybind11' 2025-03-14T04:30:32.6751637Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-03-14T04:30:32.6771473Z Entering 'third_party/python-peachpy' 2025-03-14T04:30:32.6824676Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-03-14T04:30:32.6839831Z Entering 'third_party/sleef' 2025-03-14T04:30:32.6893602Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-03-14T04:30:32.6910374Z Entering 'third_party/tensorpipe' 2025-03-14T04:30:32.6961192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-03-14T04:30:32.6977591Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T04:30:32.7031556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-03-14T04:30:32.7049408Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T04:30:32.7099978Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-03-14T04:30:32.7115712Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T04:30:32.7168412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-03-14T04:30:32.7186492Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T04:30:32.7235774Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-03-14T04:30:32.7252555Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T04:30:32.7305174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-03-14T04:30:32.7814261Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-03-14T04:30:32.8131732Z Entering 'android/libs/fbjni' 2025-03-14T04:30:32.8172699Z Entering 'third_party/FP16' 2025-03-14T04:30:32.8216658Z Entering 'third_party/FXdiv' 2025-03-14T04:30:32.8260146Z Entering 'third_party/NNPACK' 2025-03-14T04:30:32.8304660Z Entering 'third_party/NVTX' 2025-03-14T04:30:32.8349572Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T04:30:32.8393538Z Entering 'third_party/XNNPACK' 2025-03-14T04:30:32.8454858Z Entering 'third_party/benchmark' 2025-03-14T04:30:32.8499179Z Entering 'third_party/composable_kernel' 2025-03-14T04:30:32.8549997Z Entering 'third_party/cpp-httplib' 2025-03-14T04:30:32.8594553Z Entering 'third_party/cpuinfo' 2025-03-14T04:30:32.8636155Z Entering 'third_party/cudnn_frontend' 2025-03-14T04:30:32.8679472Z Entering 'third_party/cutlass' 2025-03-14T04:30:32.8741259Z Entering 'third_party/eigen' 2025-03-14T04:30:32.8790187Z Entering 'third_party/fbgemm' 2025-03-14T04:30:32.8835691Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T04:30:32.8876571Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T04:30:32.8919117Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T04:30:32.8972602Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T04:30:32.9017272Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T04:30:32.9061259Z Entering 'third_party/flash-attention' 2025-03-14T04:30:32.9107958Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T04:30:32.9167891Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T04:30:32.9222513Z Entering 'third_party/flatbuffers' 2025-03-14T04:30:32.9269100Z Entering 'third_party/fmt' 2025-03-14T04:30:32.9314822Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T04:30:32.9357711Z Entering 'third_party/gloo' 2025-03-14T04:30:32.9402726Z Entering 'third_party/googletest' 2025-03-14T04:30:32.9448312Z Entering 'third_party/ideep' 2025-03-14T04:30:32.9491743Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T04:30:32.9543037Z Entering 'third_party/ittapi' 2025-03-14T04:30:32.9587613Z Entering 'third_party/kineto' 2025-03-14T04:30:32.9631412Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T04:30:32.9673459Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T04:30:32.9716508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T04:30:32.9758913Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T04:30:32.9803671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T04:30:32.9845484Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T04:30:32.9891912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T04:30:32.9932870Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T04:30:32.9973538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T04:30:33.0018853Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T04:30:33.0062948Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T04:30:33.0109793Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T04:30:33.0155462Z Entering 'third_party/kleidiai' 2025-03-14T04:30:33.0201434Z Entering 'third_party/mimalloc' 2025-03-14T04:30:33.0244368Z Entering 'third_party/nlohmann' 2025-03-14T04:30:33.0292814Z Entering 'third_party/onnx' 2025-03-14T04:30:33.0353205Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T04:30:33.0397390Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T04:30:33.0443153Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T04:30:33.0487290Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T04:30:33.0529967Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T04:30:33.0571461Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T04:30:33.0616315Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T04:30:33.0658584Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T04:30:33.0701573Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T04:30:33.0743911Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T04:30:33.0790374Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T04:30:33.0836313Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T04:30:33.0909887Z Entering 'third_party/pocketfft' 2025-03-14T04:30:33.0953835Z Entering 'third_party/protobuf' 2025-03-14T04:30:33.0999782Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T04:30:33.1052898Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T04:30:33.1101182Z Entering 'third_party/psimd' 2025-03-14T04:30:33.1146909Z Entering 'third_party/pthreadpool' 2025-03-14T04:30:33.1191646Z Entering 'third_party/pybind11' 2025-03-14T04:30:33.1234570Z Entering 'third_party/python-peachpy' 2025-03-14T04:30:33.1277939Z Entering 'third_party/sleef' 2025-03-14T04:30:33.1320921Z Entering 'third_party/tensorpipe' 2025-03-14T04:30:33.1364585Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T04:30:33.1409686Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T04:30:33.1450734Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T04:30:33.1494006Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T04:30:33.1535590Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T04:30:33.1595698Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-03-14T04:30:33.1905554Z Entering 'android/libs/fbjni' 2025-03-14T04:30:33.1949371Z Entering 'third_party/FP16' 2025-03-14T04:30:33.1994565Z Entering 'third_party/FXdiv' 2025-03-14T04:30:33.2035703Z Entering 'third_party/NNPACK' 2025-03-14T04:30:33.2080240Z Entering 'third_party/NVTX' 2025-03-14T04:30:33.2123328Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T04:30:33.2169384Z Entering 'third_party/XNNPACK' 2025-03-14T04:30:33.2229837Z Entering 'third_party/benchmark' 2025-03-14T04:30:33.2277026Z Entering 'third_party/composable_kernel' 2025-03-14T04:30:33.2330956Z Entering 'third_party/cpp-httplib' 2025-03-14T04:30:33.2373435Z Entering 'third_party/cpuinfo' 2025-03-14T04:30:33.2418021Z Entering 'third_party/cudnn_frontend' 2025-03-14T04:30:33.2463179Z Entering 'third_party/cutlass' 2025-03-14T04:30:33.2515721Z Entering 'third_party/eigen' 2025-03-14T04:30:33.2561261Z Entering 'third_party/fbgemm' 2025-03-14T04:30:33.2612865Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T04:30:33.2654442Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T04:30:33.2697493Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T04:30:33.2747432Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T04:30:33.2790487Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T04:30:33.2835562Z Entering 'third_party/flash-attention' 2025-03-14T04:30:33.2880306Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T04:30:33.2930952Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T04:30:33.2984795Z Entering 'third_party/flatbuffers' 2025-03-14T04:30:33.3032381Z Entering 'third_party/fmt' 2025-03-14T04:30:33.3075760Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T04:30:33.3119203Z Entering 'third_party/gloo' 2025-03-14T04:30:33.3162595Z Entering 'third_party/googletest' 2025-03-14T04:30:33.3209610Z Entering 'third_party/ideep' 2025-03-14T04:30:33.3252140Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T04:30:33.3303044Z Entering 'third_party/ittapi' 2025-03-14T04:30:33.3346688Z Entering 'third_party/kineto' 2025-03-14T04:30:33.3392743Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T04:30:33.3434871Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T04:30:33.3480108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T04:30:33.3522482Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T04:30:33.3564570Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T04:30:33.3609839Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T04:30:33.3654487Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T04:30:33.3697365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T04:30:33.3740407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T04:30:33.3784824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T04:30:33.3833358Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T04:30:33.3873344Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T04:30:33.3916084Z Entering 'third_party/kleidiai' 2025-03-14T04:30:33.3959736Z Entering 'third_party/mimalloc' 2025-03-14T04:30:33.4006621Z Entering 'third_party/nlohmann' 2025-03-14T04:30:33.4052888Z Entering 'third_party/onnx' 2025-03-14T04:30:33.4114375Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T04:30:33.4156982Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T04:30:33.4203781Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T04:30:33.4251125Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T04:30:33.4292119Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T04:30:33.4334166Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T04:30:33.4377150Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T04:30:33.4419967Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T04:30:33.4462340Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T04:30:33.4506445Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T04:30:33.4550845Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T04:30:33.4595051Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T04:30:33.4661894Z Entering 'third_party/pocketfft' 2025-03-14T04:30:33.4707273Z Entering 'third_party/protobuf' 2025-03-14T04:30:33.4754882Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T04:30:33.4796435Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T04:30:33.4840331Z Entering 'third_party/psimd' 2025-03-14T04:30:33.4887170Z Entering 'third_party/pthreadpool' 2025-03-14T04:30:33.4930054Z Entering 'third_party/pybind11' 2025-03-14T04:30:33.4972152Z Entering 'third_party/python-peachpy' 2025-03-14T04:30:33.5015710Z Entering 'third_party/sleef' 2025-03-14T04:30:33.5059577Z Entering 'third_party/tensorpipe' 2025-03-14T04:30:33.5108331Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T04:30:33.5158420Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T04:30:33.5201651Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T04:30:33.5245174Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T04:30:33.5289265Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T04:30:33.5347434Z ##[endgroup] 2025-03-14T04:30:33.5388441Z [command]/usr/bin/git log -1 --format=%H 2025-03-14T04:30:33.5413840Z aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:30:33.5600300Z Prepare all required actions 2025-03-14T04:30:33.5600983Z Getting action download info 2025-03-14T04:30:33.6691211Z ##[group]Run ./.github/actions/setup-linux 2025-03-14T04:30:33.6691582Z env: 2025-03-14T04:30:33.6691841Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:33.6692139Z ##[endgroup] 2025-03-14T04:30:33.6737940Z ##[group]Run set -euo pipefail 2025-03-14T04:30:33.6738334Z set -euo pipefail 2025-03-14T04:30:33.6738676Z function get_ec2_metadata() { 2025-03-14T04:30:33.6739107Z  # Pulled from instance metadata endpoint for EC2 2025-03-14T04:30:33.6739803Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-03-14T04:30:33.6740423Z  category=$1 2025-03-14T04:30:33.6740832Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-03-14T04:30:33.6741325Z  runner_name_str=i-0f8cc103977fc9a05 2025-03-14T04:30:33.6741775Z  if [[ -f /.inarc ]]; then 2025-03-14T04:30:33.6742169Z  echo "ARC Runner, no info on ec2 metadata" 2025-03-14T04:30:33.6742756Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-03-14T04:30:33.6743278Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-03-14T04:30:33.6743760Z  else 2025-03-14T04:30:33.6744686Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-03-14T04:30:33.6745669Z  fi 2025-03-14T04:30:33.6745927Z } 2025-03-14T04:30:33.6746231Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-03-14T04:30:33.6746706Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-03-14T04:30:33.6747224Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-03-14T04:30:33.6747691Z echo "system info $(uname -a)" 2025-03-14T04:30:33.6754277Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:33.6754773Z env: 2025-03-14T04:30:33.6755060Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:33.6755382Z ##[endgroup] 2025-03-14T04:30:33.6901047Z ami-id: ami-05b10e08d247fb927 2025-03-14T04:30:33.7013580Z instance-id: i-0f8cc103977fc9a05 2025-03-14T04:30:33.7118775Z instance-type: g4dn.12xlarge 2025-03-14T04:30:33.7129581Z system info Linux ip-10-1-65-155.ec2.internal 6.1.128-136.201.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Feb 10 16:18:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-03-14T04:30:33.7180485Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T04:30:33.7181715Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T04:30:33.7188175Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:33.7188629Z env: 2025-03-14T04:30:33.7188905Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:33.7189329Z ##[endgroup] 2025-03-14T04:30:33.7251802Z ##[group]Run if systemctl is-active --quiet docker; then 2025-03-14T04:30:33.7252315Z if systemctl is-active --quiet docker; then 2025-03-14T04:30:33.7252774Z  echo "Docker daemon is running..."; 2025-03-14T04:30:33.7253157Z else 2025-03-14T04:30:33.7253568Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2025-03-14T04:30:33.7254161Z fi 2025-03-14T04:30:33.7259536Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:33.7259958Z env: 2025-03-14T04:30:33.7260200Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:33.7260492Z ##[endgroup] 2025-03-14T04:30:33.7331367Z Docker daemon is running... 2025-03-14T04:30:33.7377836Z ##[group]Run nick-fields/retry@v3.0.0 2025-03-14T04:30:33.7378215Z with: 2025-03-14T04:30:33.7378477Z shell: bash 2025-03-14T04:30:33.7378912Z timeout_minutes: 5 2025-03-14T04:30:33.7379224Z max_attempts: 3 2025-03-14T04:30:33.7379541Z retry_wait_seconds: 30 2025-03-14T04:30:33.7382447Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-03-14T04:30:33.7384888Z polling_interval_seconds: 1 2025-03-14T04:30:33.7385216Z warning_on_retry: true 2025-03-14T04:30:33.7385520Z continue_on_error: false 2025-03-14T04:30:33.7385813Z env: 2025-03-14T04:30:33.7386066Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:33.7386361Z AWS_RETRY_MODE: standard 2025-03-14T04:30:33.7386745Z AWS_MAX_ATTEMPTS: 5 2025-03-14T04:30:33.7387034Z AWS_DEFAULT_REGION: us-east-1 2025-03-14T04:30:33.7387342Z ##[endgroup] 2025-03-14T04:30:34.9153497Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-03-14T04:30:34.9154468Z Configure a credential helper to remove this warning. See 2025-03-14T04:30:34.9155188Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-03-14T04:30:34.9155683Z 2025-03-14T04:30:34.9155816Z Login Succeeded 2025-03-14T04:30:35.4551650Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-03-14T04:30:35.4552439Z Configure a credential helper to remove this warning. See 2025-03-14T04:30:35.4553510Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-03-14T04:30:35.4554003Z 2025-03-14T04:30:35.4554120Z Login Succeeded 2025-03-14T04:30:35.8303441Z Command completed after 1 attempt(s). 2025-03-14T04:30:35.8375533Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T04:30:35.8376328Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T04:30:35.8377014Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T04:30:35.8384087Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:35.8384555Z env: 2025-03-14T04:30:35.8384841Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:35.8385179Z ##[endgroup] 2025-03-14T04:30:35.8486345Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-03-14T04:30:35.8486986Z # ignore expansion of "docker ps -q" since it could be empty 2025-03-14T04:30:35.8487479Z # shellcheck disable=SC2046 2025-03-14T04:30:35.8487887Z docker stop $(docker ps -q) || true 2025-03-14T04:30:35.8488295Z # Prune all of the docker images 2025-03-14T04:30:35.8488699Z docker system prune -af 2025-03-14T04:30:35.8494448Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:35.8494901Z env: 2025-03-14T04:30:35.8495166Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:35.8495480Z ##[endgroup] 2025-03-14T04:30:35.8788525Z "docker stop" requires at least 1 argument. 2025-03-14T04:30:35.8789036Z See 'docker stop --help'. 2025-03-14T04:30:35.8789279Z 2025-03-14T04:30:35.8789483Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-03-14T04:30:35.8789833Z 2025-03-14T04:30:35.8789976Z Stop one or more running containers 2025-03-14T04:30:35.8935106Z Total reclaimed space: 0B 2025-03-14T04:30:35.8974038Z ##[group]Run set +e 2025-03-14T04:30:35.8974390Z set +e 2025-03-14T04:30:35.8974690Z set -x 2025-03-14T04:30:35.8974987Z  2025-03-14T04:30:35.8975311Z PT_DOMAIN=download.pytorch.org 2025-03-14T04:30:35.8976026Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2025-03-14T04:30:35.8977130Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2025-03-14T04:30:35.8977816Z # one is returned at random 2025-03-14T04:30:35.8978319Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2025-03-14T04:30:35.8978801Z  2025-03-14T04:30:35.8979113Z if [ -z "${RESOLVED_IP}" ]; then 2025-03-14T04:30:35.8979663Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2025-03-14T04:30:35.8980317Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2025-03-14T04:30:35.8980813Z  2025-03-14T04:30:35.8981121Z  if [ -z "${RESOLVED_IP}" ]; then 2025-03-14T04:30:35.8981608Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2025-03-14T04:30:35.8982186Z  exit 1 2025-03-14T04:30:35.8982466Z  fi 2025-03-14T04:30:35.8982703Z fi 2025-03-14T04:30:35.8982951Z  2025-03-14T04:30:35.8983259Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2025-03-14T04:30:35.8983789Z  # Clean up any old records first 2025-03-14T04:30:35.8984202Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2025-03-14T04:30:35.8984577Z fi 2025-03-14T04:30:35.8984828Z  2025-03-14T04:30:35.8985182Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2025-03-14T04:30:35.8985634Z cat /etc/hosts 2025-03-14T04:30:35.8991158Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:35.8991582Z env: 2025-03-14T04:30:35.8991842Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:35.8992143Z ##[endgroup] 2025-03-14T04:30:35.9016859Z + PT_DOMAIN=download.pytorch.org 2025-03-14T04:30:35.9023776Z ++ dig -4 +short download.pytorch.org 2025-03-14T04:30:35.9024928Z ++ tail -n1 2025-03-14T04:30:35.9278308Z + RESOLVED_IP=18.160.10.36 2025-03-14T04:30:35.9278709Z + '[' -z 18.160.10.36 ']' 2025-03-14T04:30:35.9279106Z + grep -r download.pytorch.org /etc/hosts 2025-03-14T04:30:35.9291181Z 18.160.10.28 download.pytorch.org 2025-03-14T04:30:35.9292810Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2025-03-14T04:30:36.0246597Z + echo '18.160.10.36 download.pytorch.org' 2025-03-14T04:30:36.0247066Z + sudo tee -a /etc/hosts 2025-03-14T04:30:36.2318317Z 18.160.10.36 download.pytorch.org 2025-03-14T04:30:36.2338809Z + cat /etc/hosts 2025-03-14T04:30:36.2348206Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2025-03-14T04:30:36.2353652Z ::1 localhost6 localhost6.localdomain6 2025-03-14T04:30:36.2354196Z 18.160.10.36 download.pytorch.org 2025-03-14T04:30:36.2529209Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-03-14T04:30:36.2529736Z with: 2025-03-14T04:30:36.2530537Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:36.2531429Z docker-build-dir: .ci/docker 2025-03-14T04:30:36.2531758Z working-directory: . 2025-03-14T04:30:36.2532165Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:36.2532613Z force-push: false 2025-03-14T04:30:36.2532881Z env: 2025-03-14T04:30:36.2533132Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:36.2533419Z ##[endgroup] 2025-03-14T04:30:36.2562103Z ##[group]Run set -ex 2025-03-14T04:30:36.2562491Z set -ex 2025-03-14T04:30:36.2562772Z  2025-03-14T04:30:36.2563274Z # If the docker build directory or the build script doesn't exist, the action will 2025-03-14T04:30:36.2564157Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-03-14T04:30:36.2564859Z # job could then download the pre-built image as usual 2025-03-14T04:30:36.2565612Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2025-03-14T04:30:36.2566284Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-03-14T04:30:36.2566791Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-03-14T04:30:36.2567259Z  2025-03-14T04:30:36.2567657Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2025-03-14T04:30:36.2568150Z  exit 0 2025-03-14T04:30:36.2568413Z else 2025-03-14T04:30:36.2568703Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-03-14T04:30:36.2569074Z fi 2025-03-14T04:30:36.2569525Z  2025-03-14T04:30:36.2570191Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-03-14T04:30:36.2570919Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-03-14T04:30:36.2571567Z  # use it as it is, but first let's extract the tag 2025-03-14T04:30:36.2572155Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-03-14T04:30:36.2572767Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-03-14T04:30:36.2573357Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-03-14T04:30:36.2574015Z else 2025-03-14T04:30:36.2574403Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-03-14T04:30:36.2574961Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-03-14T04:30:36.2575724Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-03-14T04:30:36.2576392Z fi 2025-03-14T04:30:36.2582964Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:36.2583387Z env: 2025-03-14T04:30:36.2583641Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:36.2583953Z REPO_NAME: pytorch 2025-03-14T04:30:36.2584770Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:36.2585648Z DOCKER_BUILD_DIR: .ci/docker 2025-03-14T04:30:36.2586067Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:36.2586512Z ##[endgroup] 2025-03-14T04:30:36.2611177Z + [[ ! -d .ci/docker ]] 2025-03-14T04:30:36.2611578Z + [[ ! -f .ci/docker/build.sh ]] 2025-03-14T04:30:36.2611953Z + echo skip=false 2025-03-14T04:30:36.2613126Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-03-14T04:30:36.2619889Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:36.2620803Z ++ awk -F '[:,]' '{print $2}' 2025-03-14T04:30:36.2641250Z + DOCKER_TAG=aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:36.2641815Z + echo docker-tag=aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:36.2642880Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:36.2676787Z ##[group]Run set +e 2025-03-14T04:30:36.2677170Z set +e 2025-03-14T04:30:36.2677468Z set -x 2025-03-14T04:30:36.2677760Z  2025-03-14T04:30:36.2678038Z login() { 2025-03-14T04:30:36.2678629Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-03-14T04:30:36.2679273Z } 2025-03-14T04:30:36.2679548Z  2025-03-14T04:30:36.2679837Z retry () { 2025-03-14T04:30:36.2680189Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-03-14T04:30:36.2680593Z } 2025-03-14T04:30:36.2680862Z  2025-03-14T04:30:36.2681152Z retry login "${DOCKER_REGISTRY}" 2025-03-14T04:30:36.2681544Z  2025-03-14T04:30:36.2681829Z START_TIME=$(date +%s) 2025-03-14T04:30:36.2682202Z # Wait up to 120 minutes 2025-03-14T04:30:36.2682671Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-03-14T04:30:36.2683280Z  # Check if image already exists, if it does then skip building it 2025-03-14T04:30:36.2683891Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-03-14T04:30:36.2684350Z  exit 0 2025-03-14T04:30:36.2703211Z  fi 2025-03-14T04:30:36.2703570Z  2025-03-14T04:30:36.2704065Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-03-14T04:30:36.2704847Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-03-14T04:30:36.2705607Z  # latter, it will wait for the Docker images to become available before continuing 2025-03-14T04:30:36.2706205Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-03-14T04:30:36.2706671Z  # It's a Docker build job, let's build the image 2025-03-14T04:30:36.2707122Z  break 2025-03-14T04:30:36.2707600Z  else 2025-03-14T04:30:36.2707998Z  # It's a regular build job, wait for the image to become available 2025-03-14T04:30:36.2708481Z  sleep 300 2025-03-14T04:30:36.2708778Z  fi 2025-03-14T04:30:36.2709034Z done 2025-03-14T04:30:36.2709285Z  2025-03-14T04:30:36.2709690Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-03-14T04:30:36.2710502Z # be empty. The default action would be to continue rebuild the image 2025-03-14T04:30:36.2711092Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-03-14T04:30:36.2711619Z  # if we're on the base branch then use the parent commit 2025-03-14T04:30:36.2712079Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-03-14T04:30:36.2712455Z else 2025-03-14T04:30:36.2712950Z  # otherwise we're on a PR, so use the most recent base commit 2025-03-14T04:30:36.2713693Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-03-14T04:30:36.2714178Z fi 2025-03-14T04:30:36.2714445Z  2025-03-14T04:30:36.2714742Z if [[ -z "${MERGE_BASE}" ]]; then 2025-03-14T04:30:36.2715192Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-03-14T04:30:36.2715612Z  2025-03-14T04:30:36.2716188Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-03-14T04:30:36.2716862Z  exit 0 2025-03-14T04:30:36.2717314Z fi 2025-03-14T04:30:36.2717576Z  2025-03-14T04:30:36.2717953Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-03-14T04:30:36.2718778Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-03-14T04:30:36.2719488Z  exit 1 2025-03-14T04:30:36.2719773Z fi 2025-03-14T04:30:36.2720038Z  2025-03-14T04:30:36.2720488Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-03-14T04:30:36.2721287Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-03-14T04:30:36.2722005Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-03-14T04:30:36.2722825Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-03-14T04:30:36.2723793Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-03-14T04:30:36.2724498Z fi 2025-03-14T04:30:36.2724820Z  2025-03-14T04:30:36.2730248Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-03-14T04:30:36.2736139Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:36.2757657Z env: 2025-03-14T04:30:36.2758455Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:36.2759320Z DOCKER_BUILD_DIR: .ci/docker 2025-03-14T04:30:36.2760392Z BASE_REVISION: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:30:36.2762974Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:36.2765551Z DOCKER_TAG: aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:36.2766896Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:36.2767306Z DOCKER_PUSH: 2025-03-14T04:30:36.2767547Z ##[endgroup] 2025-03-14T04:30:36.2792216Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:36.2792751Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:36.2796263Z + aws ecr get-login-password --region us-east-1 2025-03-14T04:30:36.2796879Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:36.8204078Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-03-14T04:30:36.8205156Z Configure a credential helper to remove this warning. See 2025-03-14T04:30:36.8205952Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-03-14T04:30:36.8206419Z 2025-03-14T04:30:36.8206547Z Login Succeeded 2025-03-14T04:30:36.8223975Z ++ date +%s 2025-03-14T04:30:36.8232412Z + START_TIME=1741926636 2025-03-14T04:30:36.8236449Z ++ date +%s 2025-03-14T04:30:36.8245578Z + [[ 1741919436 -lt 1741926636 ]] 2025-03-14T04:30:36.8246633Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:37.1007211Z { 2025-03-14T04:30:37.1007587Z "schemaVersion": 2, 2025-03-14T04:30:37.1008115Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-03-14T04:30:37.1008661Z "config": { 2025-03-14T04:30:37.1009082Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-03-14T04:30:37.1009577Z "size": 50452, 2025-03-14T04:30:37.1010103Z "digest": "sha256:f27528c1978e671c56c46509bcb1b201a995437aea125e51d1442a668bd40735" 2025-03-14T04:30:37.1010671Z }, 2025-03-14T04:30:37.1010988Z "layers": [ 2025-03-14T04:30:37.1011315Z { 2025-03-14T04:30:37.1011727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1012241Z "size": 28580681, 2025-03-14T04:30:37.1012754Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2025-03-14T04:30:37.1013324Z }, 2025-03-14T04:30:37.1013561Z { 2025-03-14T04:30:37.1014245Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1014757Z "size": 7943353, 2025-03-14T04:30:37.1015260Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2025-03-14T04:30:37.1015829Z }, 2025-03-14T04:30:37.1016078Z { 2025-03-14T04:30:37.1016476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1016977Z "size": 56226389, 2025-03-14T04:30:37.1017488Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2025-03-14T04:30:37.1018048Z }, 2025-03-14T04:30:37.1018295Z { 2025-03-14T04:30:37.1018689Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1019187Z "size": 187, 2025-03-14T04:30:37.1019778Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2025-03-14T04:30:37.1020319Z }, 2025-03-14T04:30:37.1020557Z { 2025-03-14T04:30:37.1020939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1021608Z "size": 6885, 2025-03-14T04:30:37.1022109Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2025-03-14T04:30:37.1022733Z }, 2025-03-14T04:30:37.1023002Z { 2025-03-14T04:30:37.1023401Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1023945Z "size": 1383662475, 2025-03-14T04:30:37.1024450Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2025-03-14T04:30:37.1025001Z }, 2025-03-14T04:30:37.1025247Z { 2025-03-14T04:30:37.1025649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1026151Z "size": 62282, 2025-03-14T04:30:37.1026641Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2025-03-14T04:30:37.1027198Z }, 2025-03-14T04:30:37.1027446Z { 2025-03-14T04:30:37.1027850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1028351Z "size": 1684, 2025-03-14T04:30:37.1028847Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2025-03-14T04:30:37.1029414Z }, 2025-03-14T04:30:37.1029645Z { 2025-03-14T04:30:37.1030040Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1030536Z "size": 1522, 2025-03-14T04:30:37.1031128Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2025-03-14T04:30:37.1031809Z }, 2025-03-14T04:30:37.1032050Z { 2025-03-14T04:30:37.1032439Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1033031Z "size": 2445884909, 2025-03-14T04:30:37.1033710Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2025-03-14T04:30:37.1034282Z }, 2025-03-14T04:30:37.1034543Z { 2025-03-14T04:30:37.1034954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1035478Z "size": 85882, 2025-03-14T04:30:37.1035983Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2025-03-14T04:30:37.1036560Z }, 2025-03-14T04:30:37.1036811Z { 2025-03-14T04:30:37.1037205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1037719Z "size": 1894, 2025-03-14T04:30:37.1038228Z "digest": "sha256:d6ee0b7aac7a248d2611054390e644c9e333ad45a00e8b4d9c0d468ca843d34d" 2025-03-14T04:30:37.1038812Z }, 2025-03-14T04:30:37.1039063Z { 2025-03-14T04:30:37.1039735Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1040197Z "size": 655215498, 2025-03-14T04:30:37.1040671Z "digest": "sha256:5a2a361f1d5ffc75ae1ebdca1b3d20059f339b0c9555602c800d87a633a94294" 2025-03-14T04:30:37.1041200Z }, 2025-03-14T04:30:37.1041431Z { 2025-03-14T04:30:37.1041800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1042259Z "size": 704, 2025-03-14T04:30:37.1042791Z "digest": "sha256:9875190f952116d672ea745fea3032465e9448a0e5e27b1df20d3d676a166cda" 2025-03-14T04:30:37.1043305Z }, 2025-03-14T04:30:37.1043536Z { 2025-03-14T04:30:37.1043904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1044368Z "size": 1257, 2025-03-14T04:30:37.1044813Z "digest": "sha256:bf9091491567f3cd1ee29ba3e3c32ebad3a6751a1a3515c0268ad1e27632483c" 2025-03-14T04:30:37.1045352Z }, 2025-03-14T04:30:37.1045586Z { 2025-03-14T04:30:37.1045955Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1046421Z "size": 484, 2025-03-14T04:30:37.1046889Z "digest": "sha256:cada203b76efa095b7b5298e41177ebaf9c74ed67d4e345e05c5dfe9d17fb68f" 2025-03-14T04:30:37.1047420Z }, 2025-03-14T04:30:37.1047657Z { 2025-03-14T04:30:37.1048030Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1048501Z "size": 91726322, 2025-03-14T04:30:37.1048980Z "digest": "sha256:703d00dca2f9eb8874118de723d83ed78e196ab5b2f4e6536353057562de2317" 2025-03-14T04:30:37.1049499Z }, 2025-03-14T04:30:37.1049735Z { 2025-03-14T04:30:37.1050108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1050578Z "size": 3719, 2025-03-14T04:30:37.1051039Z "digest": "sha256:e7e82334ce20792527da1df61da313006c843842dec6bbee4eca5991f2e7013d" 2025-03-14T04:30:37.1051547Z }, 2025-03-14T04:30:37.1051786Z { 2025-03-14T04:30:37.1052163Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1052626Z "size": 1860, 2025-03-14T04:30:37.1053097Z "digest": "sha256:1d4ea74fe29a0aceb5d0a9b6b09cd16a65f3fbccde220253f5baa7a35b07315b" 2025-03-14T04:30:37.1053634Z }, 2025-03-14T04:30:37.1053862Z { 2025-03-14T04:30:37.1054229Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1054688Z "size": 700, 2025-03-14T04:30:37.1055138Z "digest": "sha256:5c57d945e77401a617a527ef203cf58a433d446c540a73ef7be6c55c495d2d27" 2025-03-14T04:30:37.1055647Z }, 2025-03-14T04:30:37.1055876Z { 2025-03-14T04:30:37.1056238Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1056696Z "size": 477, 2025-03-14T04:30:37.1057042Z + exit 0 2025-03-14T04:30:37.1057476Z "digest": "sha256:cc3d1769377187b6db7c70b8444a87ddb477bf4fb22dd1233b578b302be6e3ac" 2025-03-14T04:30:37.1057998Z }, 2025-03-14T04:30:37.1058215Z { 2025-03-14T04:30:37.1058665Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1059141Z "size": 2935016355, 2025-03-14T04:30:37.1059616Z "digest": "sha256:011b4dd699abed9b5f8a637a20c47d41b36f37666dee3565d0705b893bba384f" 2025-03-14T04:30:37.1060136Z }, 2025-03-14T04:30:37.1060365Z { 2025-03-14T04:30:37.1060733Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1061192Z "size": 380, 2025-03-14T04:30:37.1061650Z "digest": "sha256:eceeac2a64b3ade86c0073f74b4444846360846100ba945d54da0d7a33cb2011" 2025-03-14T04:30:37.1062170Z }, 2025-03-14T04:30:37.1062397Z { 2025-03-14T04:30:37.1062769Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1063228Z "size": 12919, 2025-03-14T04:30:37.1063684Z "digest": "sha256:f10d93f05898cd58de23788368efb2c30f8d6d911d447f89332680c322d37a0e" 2025-03-14T04:30:37.1064196Z }, 2025-03-14T04:30:37.1064425Z { 2025-03-14T04:30:37.1064774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1065236Z "size": 864, 2025-03-14T04:30:37.1065684Z "digest": "sha256:87b8b8bf9a960751234a2383c4090d4fd2aaa748e9eeb13451fae72a83ab75bb" 2025-03-14T04:30:37.1066202Z }, 2025-03-14T04:30:37.1066428Z { 2025-03-14T04:30:37.1066794Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1067253Z "size": 106, 2025-03-14T04:30:37.1067711Z "digest": "sha256:bc8f9c7ef80b1b65eb6dd17d136e9355f8c2bac57a0e9dbb425268a3f5d4ea9d" 2025-03-14T04:30:37.1068314Z }, 2025-03-14T04:30:37.1068548Z { 2025-03-14T04:30:37.1068916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1069779Z "size": 504, 2025-03-14T04:30:37.1070472Z "digest": "sha256:95b9c737ea76db1de6d7ba727d0f7a10389be7d5618a549226c457d2e80046bf" 2025-03-14T04:30:37.1071059Z }, 2025-03-14T04:30:37.1071314Z { 2025-03-14T04:30:37.1071724Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1072253Z "size": 121476958, 2025-03-14T04:30:37.1072765Z "digest": "sha256:bf861c26681741d1dbf6d187b7b0e5b87e4b9ed03b79bd9baad7bca595fe9e67" 2025-03-14T04:30:37.1073423Z }, 2025-03-14T04:30:37.1073677Z { 2025-03-14T04:30:37.1074086Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1074600Z "size": 109, 2025-03-14T04:30:37.1075100Z "digest": "sha256:a77a93d5402779d9d4ff8500ed222f7bf46262888cd123ee78604f5662c202c9" 2025-03-14T04:30:37.1075672Z }, 2025-03-14T04:30:37.1075923Z { 2025-03-14T04:30:37.1076335Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1076848Z "size": 489, 2025-03-14T04:30:37.1077351Z "digest": "sha256:d8c07316b7ca8031bb31783977108a6d795faef073e177c59dc9558aeb253e4b" 2025-03-14T04:30:37.1077934Z }, 2025-03-14T04:30:37.1078193Z { 2025-03-14T04:30:37.1078603Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1079117Z "size": 296, 2025-03-14T04:30:37.1079630Z "digest": "sha256:3f12a311796acfd718dc29e4c9ab042f74a5eed0c0f65f02df3fd332104ad5ca" 2025-03-14T04:30:37.1080195Z }, 2025-03-14T04:30:37.1080446Z { 2025-03-14T04:30:37.1080851Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1081365Z "size": 103, 2025-03-14T04:30:37.1081875Z "digest": "sha256:a283e8b27ec2769f96f0646df1e8e82effbdfb1ac3d9f2d2558df8df27283d55" 2025-03-14T04:30:37.1082466Z }, 2025-03-14T04:30:37.1082718Z { 2025-03-14T04:30:37.1083135Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1083654Z "size": 1474, 2025-03-14T04:30:37.1084168Z "digest": "sha256:dbd96d29d52e23dc9eb20a5b76f2ca4290f3672fe6600073f344e7ef19502335" 2025-03-14T04:30:37.1084755Z }, 2025-03-14T04:30:37.1085330Z { 2025-03-14T04:30:37.1085703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1086172Z "size": 427755214, 2025-03-14T04:30:37.1086793Z "digest": "sha256:1a47a677bb412df924ca356d279d06cf91c81e8d5635f0e81f52e32beb08983d" 2025-03-14T04:30:37.1087322Z }, 2025-03-14T04:30:37.1087539Z { 2025-03-14T04:30:37.1087913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1088380Z "size": 164, 2025-03-14T04:30:37.1088836Z "digest": "sha256:d12939062360f0bbc4f96c04f60a1d782a8711b0cf1db55cfb39af56744b8388" 2025-03-14T04:30:37.1089356Z }, 2025-03-14T04:30:37.1089592Z { 2025-03-14T04:30:37.1089969Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1090434Z "size": 802, 2025-03-14T04:30:37.1090871Z "digest": "sha256:17779673764c26c4622773ab631e86e95cf576117ff76286806968a40cd991e3" 2025-03-14T04:30:37.1091381Z }, 2025-03-14T04:30:37.1091618Z { 2025-03-14T04:30:37.1091985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1092445Z "size": 35806325, 2025-03-14T04:30:37.1092904Z "digest": "sha256:7b000c0c17245efe5b89cbf748144627cf709a35b1e9547d2145010d8008a572" 2025-03-14T04:30:37.1093415Z }, 2025-03-14T04:30:37.1093645Z { 2025-03-14T04:30:37.1093995Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1094451Z "size": 104, 2025-03-14T04:30:37.1094891Z "digest": "sha256:37a8d79516732bac57702525cce34f35b1751232a4c86392d62a80739100117b" 2025-03-14T04:30:37.1095399Z }, 2025-03-14T04:30:37.1095627Z { 2025-03-14T04:30:37.1095992Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1096577Z "size": 425, 2025-03-14T04:30:37.1097046Z "digest": "sha256:60b8a611752bb34ef1ee770e1e6bda5ce6cdb55f7e6efcda6bd8b8ab02c4f7f0" 2025-03-14T04:30:37.1097576Z }, 2025-03-14T04:30:37.1097804Z { 2025-03-14T04:30:37.1098171Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1098632Z "size": 20262162, 2025-03-14T04:30:37.1099108Z "digest": "sha256:e185ab76a0889fc33b3351ca60f9dae35a8aa9ab9d8b6fad8dcb785b5fbdd400" 2025-03-14T04:30:37.1099641Z }, 2025-03-14T04:30:37.1099867Z { 2025-03-14T04:30:37.1100234Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1100693Z "size": 644, 2025-03-14T04:30:37.1101125Z "digest": "sha256:aa265769b086942607060bb2d82fdf084ee875d4f2c15d662e6dd1501e2fe511" 2025-03-14T04:30:37.1101640Z }, 2025-03-14T04:30:37.1101869Z { 2025-03-14T04:30:37.1102236Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1102692Z "size": 700, 2025-03-14T04:30:37.1103143Z "digest": "sha256:5c57d945e77401a617a527ef203cf58a433d446c540a73ef7be6c55c495d2d27" 2025-03-14T04:30:37.1103656Z }, 2025-03-14T04:30:37.1103882Z { 2025-03-14T04:30:37.1104248Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1104707Z "size": 141, 2025-03-14T04:30:37.1105163Z "digest": "sha256:9b5bf41436cc5e36cde4116df4d07ba5f54606c69a2fdc085cd1b84aa9586235" 2025-03-14T04:30:37.1105684Z }, 2025-03-14T04:30:37.1105916Z { 2025-03-14T04:30:37.1106282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1106739Z "size": 136, 2025-03-14T04:30:37.1107187Z "digest": "sha256:e0440b1e75c34de256bfba55087340b0c8203b47d5b98e5502ce0a22c2581d41" 2025-03-14T04:30:37.1107684Z }, 2025-03-14T04:30:37.1107912Z { 2025-03-14T04:30:37.1108278Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1108737Z "size": 32, 2025-03-14T04:30:37.1109193Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T04:30:37.1109714Z }, 2025-03-14T04:30:37.1109941Z { 2025-03-14T04:30:37.1110307Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1110765Z "size": 193, 2025-03-14T04:30:37.1111220Z "digest": "sha256:26dc220a757cd028ab9eaaf726aae413cbf26d230a3bfa76d53ecc816107fe5e" 2025-03-14T04:30:37.1111748Z }, 2025-03-14T04:30:37.1111976Z { 2025-03-14T04:30:37.1112412Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1112951Z "size": 565, 2025-03-14T04:30:37.1113609Z "digest": "sha256:ca4fb7ee460a82a8bca7963d650826afb2244f674bffb61b3f346a5b04479e60" 2025-03-14T04:30:37.1114256Z }, 2025-03-14T04:30:37.1114494Z { 2025-03-14T04:30:37.1114908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1115426Z "size": 43163100, 2025-03-14T04:30:37.1115952Z "digest": "sha256:4537e5738954e5aa17fc60113abfeebfbfa61922d158bd1a6e5efde681258524" 2025-03-14T04:30:37.1116538Z }, 2025-03-14T04:30:37.1116791Z { 2025-03-14T04:30:37.1117196Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1117707Z "size": 106, 2025-03-14T04:30:37.1118216Z "digest": "sha256:f5c4cfdb6b319e6fa200e51e734ac9904b103c123a885abd76d0bfbc69f0c353" 2025-03-14T04:30:37.1118804Z }, 2025-03-14T04:30:37.1119055Z { 2025-03-14T04:30:37.1119471Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1120037Z "size": 1401, 2025-03-14T04:30:37.1120549Z "digest": "sha256:082b0db9631d225d87ddb930ea166afd3c0219c8ba1ea8e306670ed3809f1f4e" 2025-03-14T04:30:37.1121135Z }, 2025-03-14T04:30:37.1121389Z { 2025-03-14T04:30:37.1121802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1122302Z "size": 700, 2025-03-14T04:30:37.1122806Z "digest": "sha256:5c57d945e77401a617a527ef203cf58a433d446c540a73ef7be6c55c495d2d27" 2025-03-14T04:30:37.1123382Z }, 2025-03-14T04:30:37.1123721Z { 2025-03-14T04:30:37.1124137Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1124655Z "size": 140, 2025-03-14T04:30:37.1125161Z "digest": "sha256:03e517f304e985802bf68fa90be336961be9da4b914f66cae73037314904c81f" 2025-03-14T04:30:37.1125849Z }, 2025-03-14T04:30:37.1126082Z { 2025-03-14T04:30:37.1126450Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1126921Z "size": 120, 2025-03-14T04:30:37.1127382Z "digest": "sha256:e8379abecfa56fe60d4e48eebbe61f1e66298b2840d415bea6a7a825c40b663b" 2025-03-14T04:30:37.1127912Z }, 2025-03-14T04:30:37.1128142Z { 2025-03-14T04:30:37.1128506Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1128968Z "size": 6570537112, 2025-03-14T04:30:37.1129444Z "digest": "sha256:eee2ffe1f918180d2386e6633bc57cea5ee01212aecbfedbc4691f8987e95c08" 2025-03-14T04:30:37.1129951Z }, 2025-03-14T04:30:37.1130179Z { 2025-03-14T04:30:37.1130546Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1131007Z "size": 176, 2025-03-14T04:30:37.1131458Z "digest": "sha256:e2c124222dc9aa3206c3da5e490b49623546164b7acefc0a6020339b0dc680b6" 2025-03-14T04:30:37.1131975Z }, 2025-03-14T04:30:37.1132201Z { 2025-03-14T04:30:37.1132565Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1133025Z "size": 908, 2025-03-14T04:30:37.1133482Z "digest": "sha256:2117add440a2075a3f768b7e231b44d932b3b247c16ca0c1b98657edff974d62" 2025-03-14T04:30:37.1134000Z }, 2025-03-14T04:30:37.1134232Z { 2025-03-14T04:30:37.1134599Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1135058Z "size": 700, 2025-03-14T04:30:37.1135501Z "digest": "sha256:5c57d945e77401a617a527ef203cf58a433d446c540a73ef7be6c55c495d2d27" 2025-03-14T04:30:37.1136011Z }, 2025-03-14T04:30:37.1136224Z { 2025-03-14T04:30:37.1136591Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1137053Z "size": 135, 2025-03-14T04:30:37.1137493Z "digest": "sha256:b59fff3e88c48f58960674750445315384bcd773d317474607f1ecdc4d5d9878" 2025-03-14T04:30:37.1138004Z }, 2025-03-14T04:30:37.1138233Z { 2025-03-14T04:30:37.1138598Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1139059Z "size": 32, 2025-03-14T04:30:37.1139511Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T04:30:37.1140113Z }, 2025-03-14T04:30:37.1140346Z { 2025-03-14T04:30:37.1140717Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1141182Z "size": 158, 2025-03-14T04:30:37.1141638Z "digest": "sha256:ac968715f8f1ac6e27bf81d30dad483336e23ce926b1bb361e2df129b3bb0435" 2025-03-14T04:30:37.1142163Z }, 2025-03-14T04:30:37.1142392Z { 2025-03-14T04:30:37.1142741Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1143205Z "size": 1899, 2025-03-14T04:30:37.1143659Z "digest": "sha256:fafd68c92609757bd2b730aebe76c52e1b20cb911cf85d4178231892ebd782e3" 2025-03-14T04:30:37.1144179Z }, 2025-03-14T04:30:37.1144559Z { 2025-03-14T04:30:37.1144921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1145383Z "size": 196364604, 2025-03-14T04:30:37.1145853Z "digest": "sha256:9cebabdc36a7b031d35c281c9a0f8dd6d56466305f3034b88b7a8b077eb53a53" 2025-03-14T04:30:37.1146379Z }, 2025-03-14T04:30:37.1146606Z { 2025-03-14T04:30:37.1146969Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1147426Z "size": 165, 2025-03-14T04:30:37.1147880Z "digest": "sha256:5f15c6ed2fe960aad91dbbe9af42567cef1587162dc55719e3d4a39391be0b86" 2025-03-14T04:30:37.1148401Z }, 2025-03-14T04:30:37.1148625Z { 2025-03-14T04:30:37.1148990Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1149431Z "size": 7943, 2025-03-14T04:30:37.1149944Z "digest": "sha256:339b86aa93616123716fc31174917cd56e98634ad69c9a81d2ed2b6ccbd708b8" 2025-03-14T04:30:37.1150458Z }, 2025-03-14T04:30:37.1150684Z { 2025-03-14T04:30:37.1151050Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1151512Z "size": 8068, 2025-03-14T04:30:37.1151983Z "digest": "sha256:f3c8ea957036fb50f9670ecf9fecb87c12e7062474e9fdecffab431bc470387d" 2025-03-14T04:30:37.1152515Z }, 2025-03-14T04:30:37.1152745Z { 2025-03-14T04:30:37.1153191Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1153870Z "size": 303, 2025-03-14T04:30:37.1154378Z "digest": "sha256:d28088afca533f1a2ab5293f31fde55d4923ade72dd81850d141883472d168c7" 2025-03-14T04:30:37.1154989Z }, 2025-03-14T04:30:37.1155243Z { 2025-03-14T04:30:37.1155648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1156166Z "size": 7630438, 2025-03-14T04:30:37.1156677Z "digest": "sha256:72a202718fa1156e53fef3dd78e540057a39a9419368934028c9b74beb05c8a9" 2025-03-14T04:30:37.1157233Z }, 2025-03-14T04:30:37.1157491Z { 2025-03-14T04:30:37.1157898Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1158418Z "size": 108, 2025-03-14T04:30:37.1158926Z "digest": "sha256:230cf4bcf0c6fd08aabe66469749c457dce5b5326a67af1086b4ccb18282aeae" 2025-03-14T04:30:37.1159516Z }, 2025-03-14T04:30:37.1159776Z { 2025-03-14T04:30:37.1160186Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1160701Z "size": 54145652, 2025-03-14T04:30:37.1161206Z "digest": "sha256:986d8291a81015c941666f20f2ee4c65d09a121091d719242899987772a989d3" 2025-03-14T04:30:37.1161770Z }, 2025-03-14T04:30:37.1162022Z { 2025-03-14T04:30:37.1162429Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1162949Z "size": 495, 2025-03-14T04:30:37.1163474Z "digest": "sha256:f8d887fe0d51f2bcef8f0527b200bdffdbb9a8bb9ce0ce49650f8a361effea12" 2025-03-14T04:30:37.1164071Z }, 2025-03-14T04:30:37.1164309Z { 2025-03-14T04:30:37.1164716Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1165229Z "size": 1373229156, 2025-03-14T04:30:37.1165853Z "digest": "sha256:7c4a03ec6df5b7433794b4c998ab6f174d37a2520ec31b5a16b371b15225b35b" 2025-03-14T04:30:37.1166366Z }, 2025-03-14T04:30:37.1166592Z { 2025-03-14T04:30:37.1167035Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1167492Z "size": 106, 2025-03-14T04:30:37.1167937Z "digest": "sha256:4333d5676e89189f0d2bd6b17c07a4a13d2a179e951854dc93563ff957ada9a3" 2025-03-14T04:30:37.1168447Z }, 2025-03-14T04:30:37.1168672Z { 2025-03-14T04:30:37.1169036Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1169851Z "size": 613, 2025-03-14T04:30:37.1170544Z "digest": "sha256:0e20bddada780e2834c78b7ce7a1329a21928e419dc6bdcfe852a8f4e4686aa6" 2025-03-14T04:30:37.1171136Z }, 2025-03-14T04:30:37.1171390Z { 2025-03-14T04:30:37.1171783Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1172303Z "size": 36532329, 2025-03-14T04:30:37.1172822Z "digest": "sha256:c4747ef10bd20108ddc440a1d2290eedd75644f01e7b9f00c74639179430b273" 2025-03-14T04:30:37.1173396Z }, 2025-03-14T04:30:37.1173646Z { 2025-03-14T04:30:37.1174051Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1174566Z "size": 111, 2025-03-14T04:30:37.1175077Z "digest": "sha256:6ba2d9998c50b151d12cbba2fa97c3489d1c179c16ee9ff094bbcabd1958a04b" 2025-03-14T04:30:37.1175663Z }, 2025-03-14T04:30:37.1175913Z { 2025-03-14T04:30:37.1176320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1176834Z "size": 531, 2025-03-14T04:30:37.1177337Z "digest": "sha256:2567c336efb2c285190b57acf7518b874d7a5ebc668f2bdc297b04d9abeb49ba" 2025-03-14T04:30:37.1177916Z }, 2025-03-14T04:30:37.1178297Z { 2025-03-14T04:30:37.1178711Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1179229Z "size": 26035, 2025-03-14T04:30:37.1179729Z "digest": "sha256:93df64f3e8e204a0a645386d4e4aab6913b2d9d865eb799de593300515ce45eb" 2025-03-14T04:30:37.1180306Z }, 2025-03-14T04:30:37.1180560Z { 2025-03-14T04:30:37.1180972Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1181487Z "size": 106, 2025-03-14T04:30:37.1182212Z "digest": "sha256:2cc971e9930785366d77c450f4b99d3455ab2ffdbad9c8abc314a64ca409e56c" 2025-03-14T04:30:37.1182726Z }, 2025-03-14T04:30:37.1182956Z { 2025-03-14T04:30:37.1183318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1183775Z "size": 32, 2025-03-14T04:30:37.1184226Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T04:30:37.1184749Z }, 2025-03-14T04:30:37.1184975Z { 2025-03-14T04:30:37.1185342Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1185798Z "size": 32, 2025-03-14T04:30:37.1186247Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T04:30:37.1186752Z }, 2025-03-14T04:30:37.1186975Z { 2025-03-14T04:30:37.1187334Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1187786Z "size": 32, 2025-03-14T04:30:37.1188239Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T04:30:37.1188755Z }, 2025-03-14T04:30:37.1188979Z { 2025-03-14T04:30:37.1189342Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T04:30:37.1189799Z "size": 32, 2025-03-14T04:30:37.1190254Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T04:30:37.1190774Z } 2025-03-14T04:30:37.1191004Z ] 2025-03-14T04:30:37.1191235Z } 2025-03-14T04:30:37.1226579Z ##[group]Run set -eux 2025-03-14T04:30:37.1226947Z set -eux 2025-03-14T04:30:37.1227959Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2025-03-14T04:30:37.1235845Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:37.1236329Z env: 2025-03-14T04:30:37.1236618Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:37.1237098Z ##[endgroup] 2025-03-14T04:30:37.1268958Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-03-14T04:30:37.1269799Z + jq --raw-output .SecretString 2025-03-14T04:30:37.1270356Z + jq -r .docker_hub_readonly_token 2025-03-14T04:30:37.1270818Z + docker login --username pytorchbot --password-stdin 2025-03-14T04:30:37.7250378Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-03-14T04:30:37.7251141Z Configure a credential helper to remove this warning. See 2025-03-14T04:30:37.7252470Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-03-14T04:30:37.7252941Z 2025-03-14T04:30:37.7253059Z Login Succeeded 2025-03-14T04:30:37.7352426Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2025-03-14T04:30:37.7353109Z tag=${ECR_DOCKER_IMAGE##*/} 2025-03-14T04:30:37.7353752Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-03-14T04:30:37.7360289Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:37.7360776Z env: 2025-03-14T04:30:37.7361057Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:37.7361988Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:37.7362947Z ##[endgroup] 2025-03-14T04:30:37.7403923Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:37.7458059Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-03-14T04:30:37.7458540Z with: 2025-03-14T04:30:37.7459296Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:37.7460244Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:37.7460672Z env: 2025-03-14T04:30:37.7460918Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:37.7461235Z ##[endgroup] 2025-03-14T04:30:37.7481265Z ##[group]Run set -x 2025-03-14T04:30:37.7481604Z set -x 2025-03-14T04:30:37.7481897Z set +e 2025-03-14T04:30:37.7482185Z  2025-03-14T04:30:37.7482459Z login() { 2025-03-14T04:30:37.7483047Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-03-14T04:30:37.7483682Z } 2025-03-14T04:30:37.7483950Z  2025-03-14T04:30:37.7484270Z retry () { 2025-03-14T04:30:37.7484724Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-03-14T04:30:37.7485358Z } 2025-03-14T04:30:37.7485601Z  2025-03-14T04:30:37.7485871Z retry login "${DOCKER_REGISTRY}" 2025-03-14T04:30:37.7486217Z  2025-03-14T04:30:37.7486456Z set -e 2025-03-14T04:30:37.7486824Z # ignore output since only exit code is used for conditional 2025-03-14T04:30:37.7487363Z # only pull docker image if it's not available locally 2025-03-14T04:30:37.7487964Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-03-14T04:30:37.7488518Z  retry docker pull "${DOCKER_IMAGE}" 2025-03-14T04:30:37.7488873Z fi 2025-03-14T04:30:37.7494012Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:30:37.7494423Z env: 2025-03-14T04:30:37.7494675Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:30:37.7495491Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:37.7496433Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:37.7496862Z ##[endgroup] 2025-03-14T04:30:37.7520256Z + set +e 2025-03-14T04:30:37.7520681Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:37.7521220Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:37.7523651Z + aws ecr get-login-password --region us-east-1 2025-03-14T04:30:37.7524903Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T04:30:38.2894663Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-03-14T04:30:38.2895408Z Configure a credential helper to remove this warning. See 2025-03-14T04:30:38.2896092Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-03-14T04:30:38.2896556Z 2025-03-14T04:30:38.2898298Z Login Succeeded 2025-03-14T04:30:38.2911955Z + set -e 2025-03-14T04:30:38.2913143Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:38.3045250Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:38.3046962Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:30:38.5909192Z aa89d6e739080d90fa18625d57297c6734465849: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2025-03-14T04:30:38.5910638Z 7a2c55901189: Pulling fs layer 2025-03-14T04:30:38.5911035Z 09d415c238d7: Pulling fs layer 2025-03-14T04:30:38.5911522Z 9fe6e2e61518: Pulling fs layer 2025-03-14T04:30:38.5912049Z 41f16248e682: Pulling fs layer 2025-03-14T04:30:38.5912955Z 95d7b7817039: Pulling fs layer 2025-03-14T04:30:38.5913315Z 8f6c90485347: Pulling fs layer 2025-03-14T04:30:38.5913665Z ab17245097e4: Pulling fs layer 2025-03-14T04:30:38.5914206Z dfecd7e9912b: Pulling fs layer 2025-03-14T04:30:38.5914562Z 464a8f745445: Pulling fs layer 2025-03-14T04:30:38.5914915Z d67c111fa588: Pulling fs layer 2025-03-14T04:30:38.5915273Z 0723abde19c0: Pulling fs layer 2025-03-14T04:30:38.5915637Z d6ee0b7aac7a: Pulling fs layer 2025-03-14T04:30:38.5916027Z 5a2a361f1d5f: Pulling fs layer 2025-03-14T04:30:38.5916401Z 9875190f9521: Pulling fs layer 2025-03-14T04:30:38.5916754Z bf9091491567: Pulling fs layer 2025-03-14T04:30:38.5917113Z cada203b76ef: Pulling fs layer 2025-03-14T04:30:38.5917471Z 703d00dca2f9: Pulling fs layer 2025-03-14T04:30:38.5917799Z ab17245097e4: Waiting 2025-03-14T04:30:38.5918121Z e7e82334ce20: Pulling fs layer 2025-03-14T04:30:38.5918467Z dfecd7e9912b: Waiting 2025-03-14T04:30:38.5918947Z 1d4ea74fe29a: Pulling fs layer 2025-03-14T04:30:38.5919289Z 95d7b7817039: Waiting 2025-03-14T04:30:38.5919594Z 5c57d945e774: Pulling fs layer 2025-03-14T04:30:38.5919938Z cc3d17693771: Pulling fs layer 2025-03-14T04:30:38.5920284Z 011b4dd699ab: Pulling fs layer 2025-03-14T04:30:38.5920636Z eceeac2a64b3: Pulling fs layer 2025-03-14T04:30:38.5920987Z f10d93f05898: Pulling fs layer 2025-03-14T04:30:38.5921335Z 87b8b8bf9a96: Pulling fs layer 2025-03-14T04:30:38.5921686Z bc8f9c7ef80b: Pulling fs layer 2025-03-14T04:30:38.5922031Z 95b9c737ea76: Pulling fs layer 2025-03-14T04:30:38.5922382Z bf861c266817: Pulling fs layer 2025-03-14T04:30:38.5922732Z a77a93d54027: Pulling fs layer 2025-03-14T04:30:38.5923062Z d8c07316b7ca: Pulling fs layer 2025-03-14T04:30:38.5923395Z 464a8f745445: Waiting 2025-03-14T04:30:38.5923702Z 3f12a311796a: Pulling fs layer 2025-03-14T04:30:38.5924048Z a283e8b27ec2: Pulling fs layer 2025-03-14T04:30:38.5924396Z dbd96d29d52e: Pulling fs layer 2025-03-14T04:30:38.5924739Z 1a47a677bb41: Pulling fs layer 2025-03-14T04:30:38.5925133Z d67c111fa588: Waiting 2025-03-14T04:30:38.5925449Z d12939062360: Pulling fs layer 2025-03-14T04:30:38.5925790Z 17779673764c: Pulling fs layer 2025-03-14T04:30:38.5926134Z 7b000c0c1724: Pulling fs layer 2025-03-14T04:30:38.5926540Z 37a8d7951673: Pulling fs layer 2025-03-14T04:30:38.5926873Z bf9091491567: Waiting 2025-03-14T04:30:38.5927181Z 60b8a611752b: Pulling fs layer 2025-03-14T04:30:38.5927523Z e185ab76a088: Pulling fs layer 2025-03-14T04:30:38.5927856Z 0723abde19c0: Waiting 2025-03-14T04:30:38.5928151Z aa265769b086: Pulling fs layer 2025-03-14T04:30:38.5928633Z 703d00dca2f9: Waiting 2025-03-14T04:30:38.5928940Z cada203b76ef: Waiting 2025-03-14T04:30:38.5929296Z d6ee0b7aac7a: Waiting 2025-03-14T04:30:38.5929579Z 41f16248e682: Waiting 2025-03-14T04:30:38.5929877Z 5a2a361f1d5f: Waiting 2025-03-14T04:30:38.5930171Z 9875190f9521: Waiting 2025-03-14T04:30:38.5930463Z e7e82334ce20: Waiting 2025-03-14T04:30:38.5930759Z f10d93f05898: Waiting 2025-03-14T04:30:38.5931061Z d8c07316b7ca: Waiting 2025-03-14T04:30:38.5931362Z 1d4ea74fe29a: Waiting 2025-03-14T04:30:38.5931672Z 87b8b8bf9a96: Waiting 2025-03-14T04:30:38.5931968Z 3f12a311796a: Waiting 2025-03-14T04:30:38.5932260Z 5c57d945e774: Waiting 2025-03-14T04:30:38.5932570Z 9b5bf41436cc: Pulling fs layer 2025-03-14T04:30:38.5932909Z a283e8b27ec2: Waiting 2025-03-14T04:30:38.5933220Z e0440b1e75c3: Pulling fs layer 2025-03-14T04:30:38.5933553Z cc3d17693771: Waiting 2025-03-14T04:30:38.5933845Z bf861c266817: Waiting 2025-03-14T04:30:38.5934128Z bc8f9c7ef80b: Waiting 2025-03-14T04:30:38.5934449Z 4f4fb700ef54: Pulling fs layer 2025-03-14T04:30:38.5934793Z dbd96d29d52e: Waiting 2025-03-14T04:30:38.5935097Z 95b9c737ea76: Waiting 2025-03-14T04:30:38.5935397Z a77a93d54027: Waiting 2025-03-14T04:30:38.5935695Z 011b4dd699ab: Waiting 2025-03-14T04:30:38.5936010Z 26dc220a757c: Pulling fs layer 2025-03-14T04:30:38.5936358Z eceeac2a64b3: Waiting 2025-03-14T04:30:38.5936665Z 1a47a677bb41: Waiting 2025-03-14T04:30:38.5936987Z ca4fb7ee460a: Pulling fs layer 2025-03-14T04:30:38.5937342Z 4537e5738954: Pulling fs layer 2025-03-14T04:30:38.5937801Z d12939062360: Waiting 2025-03-14T04:30:38.5938125Z f5c4cfdb6b31: Pulling fs layer 2025-03-14T04:30:38.5938466Z aa265769b086: Waiting 2025-03-14T04:30:38.5938763Z 082b0db9631d: Pulling fs layer 2025-03-14T04:30:38.5939101Z 4f4fb700ef54: Waiting 2025-03-14T04:30:38.5939403Z e0440b1e75c3: Waiting 2025-03-14T04:30:38.5939704Z 17779673764c: Waiting 2025-03-14T04:30:38.5940002Z 60b8a611752b: Waiting 2025-03-14T04:30:38.5940317Z 03e517f304e9: Pulling fs layer 2025-03-14T04:30:38.5940657Z 9b5bf41436cc: Waiting 2025-03-14T04:30:38.5940952Z 26dc220a757c: Waiting 2025-03-14T04:30:38.5941265Z e8379abecfa5: Pulling fs layer 2025-03-14T04:30:38.5941600Z 7b000c0c1724: Waiting 2025-03-14T04:30:38.5941894Z e185ab76a088: Waiting 2025-03-14T04:30:38.5942191Z ca4fb7ee460a: Waiting 2025-03-14T04:30:38.5942509Z eee2ffe1f918: Pulling fs layer 2025-03-14T04:30:38.5942843Z 4537e5738954: Waiting 2025-03-14T04:30:38.5943132Z 37a8d7951673: Waiting 2025-03-14T04:30:38.5943426Z e2c124222dc9: Pulling fs layer 2025-03-14T04:30:38.5943765Z 082b0db9631d: Waiting 2025-03-14T04:30:38.5944075Z 2117add440a2: Pulling fs layer 2025-03-14T04:30:38.5944412Z 03e517f304e9: Waiting 2025-03-14T04:30:38.5944717Z eee2ffe1f918: Waiting 2025-03-14T04:30:38.5945020Z e8379abecfa5: Waiting 2025-03-14T04:30:38.5945323Z 2117add440a2: Waiting 2025-03-14T04:30:38.5945640Z b59fff3e88c4: Pulling fs layer 2025-03-14T04:30:38.5945992Z ac968715f8f1: Pulling fs layer 2025-03-14T04:30:38.5946323Z ac968715f8f1: Waiting 2025-03-14T04:30:38.5946643Z fafd68c92609: Pulling fs layer 2025-03-14T04:30:38.5946998Z 9cebabdc36a7: Pulling fs layer 2025-03-14T04:30:38.5947349Z 5f15c6ed2fe9: Pulling fs layer 2025-03-14T04:30:38.5947695Z 339b86aa9361: Pulling fs layer 2025-03-14T04:30:38.5948027Z f3c8ea957036: Pulling fs layer 2025-03-14T04:30:38.5948378Z d28088afca53: Pulling fs layer 2025-03-14T04:30:38.5948723Z 72a202718fa1: Pulling fs layer 2025-03-14T04:30:38.5949071Z 230cf4bcf0c6: Pulling fs layer 2025-03-14T04:30:38.5949419Z 986d8291a810: Pulling fs layer 2025-03-14T04:30:38.5949787Z fafd68c92609: Waiting 2025-03-14T04:30:38.5950092Z 9cebabdc36a7: Waiting 2025-03-14T04:30:38.5950392Z 339b86aa9361: Waiting 2025-03-14T04:30:38.5950702Z f8d887fe0d51: Pulling fs layer 2025-03-14T04:30:38.5951042Z 5f15c6ed2fe9: Waiting 2025-03-14T04:30:38.5951354Z 7c4a03ec6df5: Pulling fs layer 2025-03-14T04:30:38.5951702Z 4333d5676e89: Pulling fs layer 2025-03-14T04:30:38.5952034Z d28088afca53: Waiting 2025-03-14T04:30:38.5952344Z 0e20bddada78: Pulling fs layer 2025-03-14T04:30:38.5952784Z 986d8291a810: Waiting 2025-03-14T04:30:38.5953167Z f3c8ea957036: Waiting 2025-03-14T04:30:38.5953647Z 72a202718fa1: Waiting 2025-03-14T04:30:38.5954035Z c4747ef10bd2: Pulling fs layer 2025-03-14T04:30:38.5954384Z 230cf4bcf0c6: Waiting 2025-03-14T04:30:38.5954695Z f8d887fe0d51: Waiting 2025-03-14T04:30:38.5955010Z 6ba2d9998c50: Pulling fs layer 2025-03-14T04:30:38.5955354Z 4333d5676e89: Waiting 2025-03-14T04:30:38.5955668Z 2567c336efb2: Pulling fs layer 2025-03-14T04:30:38.5956018Z 0e20bddada78: Waiting 2025-03-14T04:30:38.5956323Z 6ba2d9998c50: Waiting 2025-03-14T04:30:38.5956643Z 93df64f3e8e2: Pulling fs layer 2025-03-14T04:30:38.5957000Z 2cc971e99307: Pulling fs layer 2025-03-14T04:30:38.5957340Z 2567c336efb2: Waiting 2025-03-14T04:30:38.5957643Z 2cc971e99307: Waiting 2025-03-14T04:30:38.9593886Z 09d415c238d7: Verifying Checksum 2025-03-14T04:30:38.9594341Z 09d415c238d7: Download complete 2025-03-14T04:30:39.0515637Z 41f16248e682: Verifying Checksum 2025-03-14T04:30:39.0516064Z 41f16248e682: Download complete 2025-03-14T04:30:39.1057370Z 7a2c55901189: Verifying Checksum 2025-03-14T04:30:39.1057870Z 7a2c55901189: Download complete 2025-03-14T04:30:39.1530909Z 95d7b7817039: Download complete 2025-03-14T04:30:39.2744030Z ab17245097e4: Verifying Checksum 2025-03-14T04:30:39.2744586Z ab17245097e4: Download complete 2025-03-14T04:30:39.3700711Z dfecd7e9912b: Download complete 2025-03-14T04:30:39.4578631Z 464a8f745445: Verifying Checksum 2025-03-14T04:30:39.4579431Z 464a8f745445: Download complete 2025-03-14T04:30:39.8105462Z 7a2c55901189: Pull complete 2025-03-14T04:30:39.9107811Z 9fe6e2e61518: Verifying Checksum 2025-03-14T04:30:39.9108302Z 9fe6e2e61518: Download complete 2025-03-14T04:30:40.0031841Z 09d415c238d7: Pull complete 2025-03-14T04:30:40.0735556Z 0723abde19c0: Verifying Checksum 2025-03-14T04:30:40.0736430Z 0723abde19c0: Download complete 2025-03-14T04:30:40.1770314Z d6ee0b7aac7a: Verifying Checksum 2025-03-14T04:30:40.1770791Z d6ee0b7aac7a: Download complete 2025-03-14T04:30:40.7495363Z 9fe6e2e61518: Pull complete 2025-03-14T04:30:40.7608343Z 41f16248e682: Pull complete 2025-03-14T04:30:40.7714062Z 95d7b7817039: Pull complete 2025-03-14T04:30:53.4352067Z 5a2a361f1d5f: Verifying Checksum 2025-03-14T04:30:53.4352775Z 5a2a361f1d5f: Download complete 2025-03-14T04:30:53.5291472Z 9875190f9521: Verifying Checksum 2025-03-14T04:30:53.5291934Z 9875190f9521: Download complete 2025-03-14T04:30:53.6258865Z bf9091491567: Verifying Checksum 2025-03-14T04:30:53.6259904Z bf9091491567: Download complete 2025-03-14T04:30:53.7146488Z cada203b76ef: Verifying Checksum 2025-03-14T04:30:53.7146954Z cada203b76ef: Download complete 2025-03-14T04:30:56.3570665Z 703d00dca2f9: Verifying Checksum 2025-03-14T04:30:56.3571127Z 703d00dca2f9: Download complete 2025-03-14T04:30:56.4861950Z e7e82334ce20: Verifying Checksum 2025-03-14T04:30:56.4862387Z e7e82334ce20: Download complete 2025-03-14T04:30:56.6010494Z 1d4ea74fe29a: Verifying Checksum 2025-03-14T04:30:56.6010932Z 1d4ea74fe29a: Download complete 2025-03-14T04:30:56.6995136Z 5c57d945e774: Download complete 2025-03-14T04:30:56.8388158Z cc3d17693771: Download complete 2025-03-14T04:31:05.5944171Z 8f6c90485347: Verifying Checksum 2025-03-14T04:31:05.5944828Z 8f6c90485347: Download complete 2025-03-14T04:31:05.6732629Z eceeac2a64b3: Verifying Checksum 2025-03-14T04:31:05.6733123Z eceeac2a64b3: Download complete 2025-03-14T04:31:05.8243464Z f10d93f05898: Verifying Checksum 2025-03-14T04:31:05.8244093Z f10d93f05898: Download complete 2025-03-14T04:31:05.9299151Z 87b8b8bf9a96: Verifying Checksum 2025-03-14T04:31:05.9300063Z 87b8b8bf9a96: Download complete 2025-03-14T04:31:06.0773146Z bc8f9c7ef80b: Verifying Checksum 2025-03-14T04:31:06.0773933Z bc8f9c7ef80b: Download complete 2025-03-14T04:31:06.1883212Z 95b9c737ea76: Download complete 2025-03-14T04:31:08.0320438Z bf861c266817: Verifying Checksum 2025-03-14T04:31:08.0320899Z bf861c266817: Download complete 2025-03-14T04:31:08.1198042Z a77a93d54027: Download complete 2025-03-14T04:31:08.2256157Z d8c07316b7ca: Download complete 2025-03-14T04:31:08.3260823Z 3f12a311796a: Download complete 2025-03-14T04:31:08.4274083Z a283e8b27ec2: Download complete 2025-03-14T04:31:08.5397686Z dbd96d29d52e: Verifying Checksum 2025-03-14T04:31:08.5398157Z dbd96d29d52e: Download complete 2025-03-14T04:31:16.2736034Z 1a47a677bb41: Verifying Checksum 2025-03-14T04:31:16.2736666Z 1a47a677bb41: Download complete 2025-03-14T04:31:16.3791986Z d12939062360: Verifying Checksum 2025-03-14T04:31:16.3792550Z d12939062360: Download complete 2025-03-14T04:31:16.4797273Z 17779673764c: Verifying Checksum 2025-03-14T04:31:16.4797702Z 17779673764c: Download complete 2025-03-14T04:31:17.3529743Z 7b000c0c1724: Verifying Checksum 2025-03-14T04:31:17.3530204Z 7b000c0c1724: Download complete 2025-03-14T04:31:17.4490069Z 37a8d7951673: Download complete 2025-03-14T04:31:17.5911505Z 60b8a611752b: Download complete 2025-03-14T04:31:18.3132234Z e185ab76a088: Verifying Checksum 2025-03-14T04:31:18.3132826Z e185ab76a088: Download complete 2025-03-14T04:31:18.4074869Z aa265769b086: Download complete 2025-03-14T04:31:18.5307997Z 9b5bf41436cc: Verifying Checksum 2025-03-14T04:31:18.5308439Z 9b5bf41436cc: Download complete 2025-03-14T04:31:18.6471836Z e0440b1e75c3: Verifying Checksum 2025-03-14T04:31:18.6472297Z e0440b1e75c3: Download complete 2025-03-14T04:31:18.6538435Z 4f4fb700ef54: Verifying Checksum 2025-03-14T04:31:18.6538851Z 4f4fb700ef54: Download complete 2025-03-14T04:31:18.7634776Z 26dc220a757c: Verifying Checksum 2025-03-14T04:31:18.7635238Z 26dc220a757c: Download complete 2025-03-14T04:31:18.8794193Z ca4fb7ee460a: Verifying Checksum 2025-03-14T04:31:18.8794660Z ca4fb7ee460a: Download complete 2025-03-14T04:31:19.7410705Z 4537e5738954: Verifying Checksum 2025-03-14T04:31:19.7411239Z 4537e5738954: Download complete 2025-03-14T04:31:19.8320217Z f5c4cfdb6b31: Download complete 2025-03-14T04:31:19.8578500Z 8f6c90485347: Pull complete 2025-03-14T04:31:19.9162684Z 082b0db9631d: Download complete 2025-03-14T04:31:20.0446706Z 03e517f304e9: Download complete 2025-03-14T04:31:20.1742879Z e8379abecfa5: Download complete 2025-03-14T04:31:20.3202967Z ab17245097e4: Pull complete 2025-03-14T04:31:20.7883624Z dfecd7e9912b: Pull complete 2025-03-14T04:31:21.2229233Z 464a8f745445: Pull complete 2025-03-14T04:31:28.6313754Z d67c111fa588: Verifying Checksum 2025-03-14T04:31:28.6314231Z d67c111fa588: Download complete 2025-03-14T04:31:28.7233931Z e2c124222dc9: Verifying Checksum 2025-03-14T04:31:28.7234371Z e2c124222dc9: Download complete 2025-03-14T04:31:28.8111962Z 2117add440a2: Download complete 2025-03-14T04:31:28.9158192Z b59fff3e88c4: Verifying Checksum 2025-03-14T04:31:28.9158644Z b59fff3e88c4: Download complete 2025-03-14T04:31:29.0138129Z ac968715f8f1: Download complete 2025-03-14T04:31:29.2129141Z fafd68c92609: Verifying Checksum 2025-03-14T04:31:29.2129732Z fafd68c92609: Download complete 2025-03-14T04:31:32.6989455Z 9cebabdc36a7: Verifying Checksum 2025-03-14T04:31:32.6989909Z 9cebabdc36a7: Download complete 2025-03-14T04:31:32.7905454Z 5f15c6ed2fe9: Verifying Checksum 2025-03-14T04:31:32.7905924Z 5f15c6ed2fe9: Download complete 2025-03-14T04:31:32.8814781Z 339b86aa9361: Verifying Checksum 2025-03-14T04:31:32.8815494Z 339b86aa9361: Download complete 2025-03-14T04:31:32.9965891Z f3c8ea957036: Download complete 2025-03-14T04:31:33.0776066Z d28088afca53: Verifying Checksum 2025-03-14T04:31:33.0776673Z d28088afca53: Download complete 2025-03-14T04:31:33.3923926Z 72a202718fa1: Verifying Checksum 2025-03-14T04:31:33.3924400Z 72a202718fa1: Download complete 2025-03-14T04:31:33.5029628Z 230cf4bcf0c6: Verifying Checksum 2025-03-14T04:31:33.5030121Z 230cf4bcf0c6: Download complete 2025-03-14T04:31:34.6509219Z 986d8291a810: Verifying Checksum 2025-03-14T04:31:34.6509650Z 986d8291a810: Download complete 2025-03-14T04:31:34.7411260Z f8d887fe0d51: Verifying Checksum 2025-03-14T04:31:34.7411711Z f8d887fe0d51: Download complete 2025-03-14T04:31:49.0261582Z 011b4dd699ab: Verifying Checksum 2025-03-14T04:31:49.0262056Z 011b4dd699ab: Download complete 2025-03-14T04:31:49.1227300Z 4333d5676e89: Verifying Checksum 2025-03-14T04:31:49.1228080Z 4333d5676e89: Download complete 2025-03-14T04:31:49.2280627Z 0e20bddada78: Verifying Checksum 2025-03-14T04:31:49.2281126Z 0e20bddada78: Download complete 2025-03-14T04:31:49.8905127Z c4747ef10bd2: Verifying Checksum 2025-03-14T04:31:49.8905609Z c4747ef10bd2: Download complete 2025-03-14T04:31:49.9957051Z 6ba2d9998c50: Verifying Checksum 2025-03-14T04:31:49.9957509Z 6ba2d9998c50: Download complete 2025-03-14T04:31:50.1181432Z 2567c336efb2: Verifying Checksum 2025-03-14T04:31:50.1181954Z 2567c336efb2: Download complete 2025-03-14T04:31:50.2409158Z 93df64f3e8e2: Verifying Checksum 2025-03-14T04:31:50.2409644Z 93df64f3e8e2: Download complete 2025-03-14T04:31:50.3433280Z 2cc971e99307: Download complete 2025-03-14T04:31:54.0293713Z d67c111fa588: Pull complete 2025-03-14T04:31:54.5027825Z 0723abde19c0: Pull complete 2025-03-14T04:31:54.9472556Z d6ee0b7aac7a: Pull complete 2025-03-14T04:31:59.3476522Z 7c4a03ec6df5: Verifying Checksum 2025-03-14T04:31:59.3476966Z 7c4a03ec6df5: Download complete 2025-03-14T04:32:04.5012179Z 5a2a361f1d5f: Pull complete 2025-03-14T04:32:04.9360141Z 9875190f9521: Pull complete 2025-03-14T04:32:05.3966294Z bf9091491567: Pull complete 2025-03-14T04:32:05.8563988Z cada203b76ef: Pull complete 2025-03-14T04:32:08.1768144Z 703d00dca2f9: Pull complete 2025-03-14T04:32:08.5936347Z e7e82334ce20: Pull complete 2025-03-14T04:32:08.9808532Z 1d4ea74fe29a: Pull complete 2025-03-14T04:32:09.3774501Z 5c57d945e774: Pull complete 2025-03-14T04:32:09.8206877Z cc3d17693771: Pull complete 2025-03-14T04:32:52.6117293Z 011b4dd699ab: Pull complete 2025-03-14T04:32:53.0789988Z eceeac2a64b3: Pull complete 2025-03-14T04:32:53.5508868Z f10d93f05898: Pull complete 2025-03-14T04:32:53.9985856Z 87b8b8bf9a96: Pull complete 2025-03-14T04:32:54.4251531Z bc8f9c7ef80b: Pull complete 2025-03-14T04:32:54.9178194Z 95b9c737ea76: Pull complete 2025-03-14T04:32:58.0397902Z bf861c266817: Pull complete 2025-03-14T04:32:58.4504084Z a77a93d54027: Pull complete 2025-03-14T04:32:58.9595952Z d8c07316b7ca: Pull complete 2025-03-14T04:32:59.4157453Z 3f12a311796a: Pull complete 2025-03-14T04:32:59.8812114Z a283e8b27ec2: Pull complete 2025-03-14T04:33:00.3359611Z dbd96d29d52e: Pull complete 2025-03-14T04:33:07.7028022Z 1a47a677bb41: Pull complete 2025-03-14T04:33:08.1062863Z d12939062360: Pull complete 2025-03-14T04:33:08.6019049Z 17779673764c: Pull complete 2025-03-14T04:33:09.7036910Z 7b000c0c1724: Pull complete 2025-03-14T04:33:10.1618998Z 37a8d7951673: Pull complete 2025-03-14T04:33:10.6352365Z 60b8a611752b: Pull complete 2025-03-14T04:33:11.3126092Z e185ab76a088: Pull complete 2025-03-14T04:33:11.7936722Z aa265769b086: Pull complete 2025-03-14T04:33:12.7940767Z 9b5bf41436cc: Pull complete 2025-03-14T04:33:13.2712220Z e0440b1e75c3: Pull complete 2025-03-14T04:33:13.6899787Z 4f4fb700ef54: Pull complete 2025-03-14T04:33:14.1518745Z 26dc220a757c: Pull complete 2025-03-14T04:33:14.6335468Z ca4fb7ee460a: Pull complete 2025-03-14T04:33:16.6665500Z 4537e5738954: Pull complete 2025-03-14T04:33:17.1652094Z f5c4cfdb6b31: Pull complete 2025-03-14T04:33:17.6263729Z 082b0db9631d: Pull complete 2025-03-14T04:33:18.5408856Z 03e517f304e9: Pull complete 2025-03-14T04:33:18.9979770Z e8379abecfa5: Pull complete 2025-03-14T04:33:20.8236227Z eee2ffe1f918: Verifying Checksum 2025-03-14T04:33:20.8236726Z eee2ffe1f918: Download complete 2025-03-14T04:35:01.9400267Z eee2ffe1f918: Pull complete 2025-03-14T04:35:02.3930164Z e2c124222dc9: Pull complete 2025-03-14T04:35:02.8301022Z 2117add440a2: Pull complete 2025-03-14T04:35:03.6314250Z b59fff3e88c4: Pull complete 2025-03-14T04:35:04.3320706Z ac968715f8f1: Pull complete 2025-03-14T04:35:04.7587740Z fafd68c92609: Pull complete 2025-03-14T04:35:10.5879179Z 9cebabdc36a7: Pull complete 2025-03-14T04:35:11.0754584Z 5f15c6ed2fe9: Pull complete 2025-03-14T04:35:11.5051332Z 339b86aa9361: Pull complete 2025-03-14T04:35:11.9624805Z f3c8ea957036: Pull complete 2025-03-14T04:35:12.4252816Z d28088afca53: Pull complete 2025-03-14T04:35:13.6965871Z 72a202718fa1: Pull complete 2025-03-14T04:35:14.1456804Z 230cf4bcf0c6: Pull complete 2025-03-14T04:35:16.0315399Z 986d8291a810: Pull complete 2025-03-14T04:35:16.4697791Z f8d887fe0d51: Pull complete 2025-03-14T04:35:29.1597318Z 7c4a03ec6df5: Pull complete 2025-03-14T04:35:29.6275284Z 4333d5676e89: Pull complete 2025-03-14T04:35:30.0754828Z 0e20bddada78: Pull complete 2025-03-14T04:35:30.9755282Z c4747ef10bd2: Pull complete 2025-03-14T04:35:31.4254858Z 6ba2d9998c50: Pull complete 2025-03-14T04:35:31.8743133Z 2567c336efb2: Pull complete 2025-03-14T04:35:32.3914067Z 93df64f3e8e2: Pull complete 2025-03-14T04:35:32.6019031Z 2cc971e99307: Pull complete 2025-03-14T04:35:32.6612140Z Digest: sha256:af572d83968aa426e5a9b40a69e9c34e3dd00ecdc49b56285f110770c0ab265b 2025-03-14T04:35:32.6640854Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:35:32.6669950Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:35:32.6722364Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T04:35:32.6723582Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T04:35:32.6732173Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:35:32.6732637Z env: 2025-03-14T04:35:32.6732915Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:35:32.6733241Z ##[endgroup] 2025-03-14T04:35:32.6903729Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2025-03-14T04:35:32.6904212Z with: 2025-03-14T04:35:32.6904471Z driver-version: 550.54.15 2025-03-14T04:35:32.6904779Z env: 2025-03-14T04:35:32.6905029Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:35:32.6905339Z ##[endgroup] 2025-03-14T04:35:32.6947562Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2025-03-14T04:35:32.6948065Z with: 2025-03-14T04:35:32.6948329Z timeout_minutes: 10 2025-03-14T04:35:32.6948625Z max_attempts: 3 2025-03-14T04:35:32.6979843Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2025-03-14T04:35:32.7010866Z retry_wait_seconds: 10 2025-03-14T04:35:32.7011210Z polling_interval_seconds: 1 2025-03-14T04:35:32.7011559Z warning_on_retry: true 2025-03-14T04:35:32.7011888Z continue_on_error: false 2025-03-14T04:35:32.7012204Z env: 2025-03-14T04:35:32.7012468Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:35:32.7012799Z DRIVER_VERSION: 550.54.15 2025-03-14T04:35:32.7013188Z ##[endgroup] 2025-03-14T04:35:32.7740722Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2025-03-14T04:35:32.7742385Z + pre_install_nvidia_driver_amzn2 2025-03-14T04:35:32.7743117Z + sudo yum remove -y nvidia-driver-latest-dkms 2025-03-14T04:35:33.2171709Z No match for argument: nvidia-driver-latest-dkms 2025-03-14T04:35:33.2172218Z No packages marked for removal. 2025-03-14T04:35:33.2237317Z Dependencies resolved. 2025-03-14T04:35:33.2246777Z Nothing to do. 2025-03-14T04:35:33.2247469Z Complete! 2025-03-14T04:35:33.2575443Z + install_nvidia_driver_common 2025-03-14T04:35:33.2576753Z + echo 'Before installing NVIDIA driver' 2025-03-14T04:35:33.2578164Z + lspci 2025-03-14T04:35:33.2579883Z Before installing NVIDIA driver 2025-03-14T04:35:33.2707404Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-03-14T04:35:33.2708085Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-03-14T04:35:33.2708803Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-03-14T04:35:33.2709507Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-03-14T04:35:33.2710124Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-03-14T04:35:33.2710805Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-03-14T04:35:33.2711446Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-03-14T04:35:33.2712042Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-03-14T04:35:33.2713006Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-03-14T04:35:33.2713790Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-03-14T04:35:33.2714448Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-03-14T04:35:33.2714979Z + lsmod 2025-03-14T04:35:33.2746409Z Module Size Used by 2025-03-14T04:35:33.2746906Z ib_core 454656 0 2025-03-14T04:35:33.2747607Z veth 36864 0 2025-03-14T04:35:33.2748082Z nvidia_modeset 1351680 0 2025-03-14T04:35:33.2748466Z video 65536 1 nvidia_modeset 2025-03-14T04:35:33.2748870Z wmi 36864 1 video 2025-03-14T04:35:33.2749231Z nvidia_uvm 4706304 0 2025-03-14T04:35:33.2749630Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2025-03-14T04:35:33.2750063Z drm 602112 1 nvidia 2025-03-14T04:35:33.2750459Z drm_panel_orientation_quirks 32768 1 drm 2025-03-14T04:35:33.2750940Z backlight 24576 3 video,drm,nvidia_modeset 2025-03-14T04:35:33.2751392Z i2c_core 110592 2 nvidia,drm 2025-03-14T04:35:33.2751763Z xt_conntrack 16384 1 2025-03-14T04:35:33.2752106Z nft_chain_nat 16384 3 2025-03-14T04:35:33.2752444Z xt_MASQUERADE 20480 1 2025-03-14T04:35:33.2752935Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-03-14T04:35:33.2753387Z nf_conntrack_netlink 57344 0 2025-03-14T04:35:33.2754081Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-03-14T04:35:33.2754662Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-03-14T04:35:33.2755083Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-03-14T04:35:33.2755479Z xfrm_user 57344 1 2025-03-14T04:35:33.2755838Z xfrm_algo 16384 1 xfrm_user 2025-03-14T04:35:33.2756226Z xt_addrtype 16384 2 2025-03-14T04:35:33.2756576Z nft_compat 20480 4 2025-03-14T04:35:33.2756988Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-03-14T04:35:33.2757535Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-03-14T04:35:33.2758035Z br_netfilter 36864 0 2025-03-14T04:35:33.2758408Z bridge 323584 1 br_netfilter 2025-03-14T04:35:33.2758809Z stp 16384 1 bridge 2025-03-14T04:35:33.2759191Z llc 16384 2 bridge,stp 2025-03-14T04:35:33.2759782Z overlay 167936 0 2025-03-14T04:35:33.2760126Z tls 135168 0 2025-03-14T04:35:33.2760463Z sunrpc 696320 1 2025-03-14T04:35:33.2760823Z nls_ascii 16384 1 2025-03-14T04:35:33.2761167Z nls_cp437 20480 1 2025-03-14T04:35:33.2761509Z vfat 24576 1 2025-03-14T04:35:33.2761850Z fat 86016 1 vfat 2025-03-14T04:35:33.2762211Z ena 180224 0 2025-03-14T04:35:33.2762544Z i8042 45056 0 2025-03-14T04:35:33.2762896Z serio 28672 3 i8042 2025-03-14T04:35:33.2763278Z skx_edac_common 28672 0 2025-03-14T04:35:33.2763612Z button 24576 0 2025-03-14T04:35:33.2763965Z ghash_clmulni_intel 16384 0 2025-03-14T04:35:33.2764326Z sch_fq_codel 20480 33 2025-03-14T04:35:33.2764679Z dm_mod 188416 0 2025-03-14T04:35:33.2765020Z fuse 163840 1 2025-03-14T04:35:33.2765367Z configfs 57344 1 2025-03-14T04:35:33.2765815Z loop 36864 0 2025-03-14T04:35:33.2766152Z dax 45056 1 dm_mod 2025-03-14T04:35:33.2766517Z dmi_sysfs 20480 0 2025-03-14T04:35:33.2766855Z crc32_pclmul 16384 0 2025-03-14T04:35:33.2767195Z crc32c_intel 24576 0 2025-03-14T04:35:33.2767530Z efivarfs 24576 1 2025-03-14T04:35:33.2767859Z + modinfo nvidia 2025-03-14T04:35:33.2768378Z filename: /lib/modules/6.1.128-136.201.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-03-14T04:35:33.2769046Z alias: char-major-195-* 2025-03-14T04:35:33.2769584Z version: 550.54.15 2025-03-14T04:35:33.2770080Z supported: external 2025-03-14T04:35:33.2770405Z license: NVIDIA 2025-03-14T04:35:33.2770744Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2025-03-14T04:35:33.2771200Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2025-03-14T04:35:33.2771631Z srcversion: 833721318DA517F0C2FEC97 2025-03-14T04:35:33.2772061Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-03-14T04:35:33.2772513Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-03-14T04:35:33.2772963Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-03-14T04:35:33.2773384Z depends: i2c-core,drm 2025-03-14T04:35:33.2773727Z retpoline: Y 2025-03-14T04:35:33.2774032Z name: nvidia 2025-03-14T04:35:33.2774514Z vermagic: 6.1.128-136.201.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-03-14T04:35:33.2775134Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-03-14T04:35:33.2775724Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-03-14T04:35:33.2776275Z parm: NVreg_ResmanDebugLevel:int 2025-03-14T04:35:33.2776683Z parm: NVreg_RmLogonRC:int 2025-03-14T04:35:33.2777085Z parm: NVreg_ModifyDeviceFiles:int 2025-03-14T04:35:33.2777508Z parm: NVreg_DeviceFileUID:int 2025-03-14T04:35:33.2777911Z parm: NVreg_DeviceFileGID:int 2025-03-14T04:35:33.2778327Z parm: NVreg_DeviceFileMode:int 2025-03-14T04:35:33.2778803Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-03-14T04:35:33.2779295Z parm: NVreg_UsePageAttributeTable:int 2025-03-14T04:35:33.2779738Z parm: NVreg_EnablePCIeGen3:int 2025-03-14T04:35:33.2780140Z parm: NVreg_EnableMSI:int 2025-03-14T04:35:33.2780529Z parm: NVreg_TCEBypassMode:int 2025-03-14T04:35:33.2780951Z parm: NVreg_EnableStreamMemOPs:int 2025-03-14T04:35:33.2781542Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-03-14T04:35:33.2782044Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-03-14T04:35:33.2782531Z parm: NVreg_EnableS0ixPowerManagement:int 2025-03-14T04:35:33.2783049Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-03-14T04:35:33.2783565Z parm: NVreg_DynamicPowerManagement:int 2025-03-14T04:35:33.2784093Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-03-14T04:35:33.2784741Z parm: NVreg_EnableGpuFirmware:int 2025-03-14T04:35:33.2785290Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-03-14T04:35:33.2785753Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-03-14T04:35:33.2786219Z parm: NVreg_EnableUserNUMAManagement:int 2025-03-14T04:35:33.2786646Z parm: NVreg_MemoryPoolSize:int 2025-03-14T04:35:33.2787051Z parm: NVreg_KMallocHeapMaxSize:int 2025-03-14T04:35:33.2787471Z parm: NVreg_VMallocHeapMaxSize:int 2025-03-14T04:35:33.2787881Z parm: NVreg_IgnoreMMIOCheck:int 2025-03-14T04:35:33.2788262Z parm: NVreg_NvLinkDisable:int 2025-03-14T04:35:33.2788689Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-03-14T04:35:33.2789137Z parm: NVreg_RegisterPCIDriver:int 2025-03-14T04:35:33.2789549Z parm: NVreg_EnableResizableBar:int 2025-03-14T04:35:33.2789968Z parm: NVreg_EnableDbgBreakpoint:int 2025-03-14T04:35:33.2790402Z parm: NVreg_EnableNonblockingOpen:int 2025-03-14T04:35:33.2790824Z parm: NVreg_RegistryDwords:charp 2025-03-14T04:35:33.2791252Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-03-14T04:35:33.2791664Z parm: NVreg_RmMsg:charp 2025-03-14T04:35:33.2792028Z parm: NVreg_GpuBlacklist:charp 2025-03-14T04:35:33.2792437Z parm: NVreg_TemporaryFilePath:charp 2025-03-14T04:35:33.2792924Z parm: NVreg_ExcludedGpus:charp 2025-03-14T04:35:33.2793509Z parm: NVreg_DmaRemapPeerMmio:int 2025-03-14T04:35:33.2794267Z parm: NVreg_RmNvlinkBandwidth:charp 2025-03-14T04:35:33.2794712Z parm: NVreg_ImexChannelCount:int 2025-03-14T04:35:33.2795131Z parm: rm_firmware_active:charp 2025-03-14T04:35:33.2795519Z + HAS_NVIDIA_DRIVER=0 2025-03-14T04:35:33.2795850Z ++ command -v nvidia-smi 2025-03-14T04:35:33.2796193Z + '[' -x /usr/bin/nvidia-smi ']' 2025-03-14T04:35:33.2796537Z + set +e 2025-03-14T04:35:33.2796950Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2025-03-14T04:35:33.3084256Z + INSTALLED_DRIVER_VERSION=550.54.15 2025-03-14T04:35:33.3084737Z + NVIDIA_SMI_STATUS=0 2025-03-14T04:35:33.3085178Z + '[' 0 -ne 0 ']' 2025-03-14T04:35:33.3085468Z + '[' 550.54.15 '!=' 550.54.15 ']' 2025-03-14T04:35:33.3085814Z + HAS_NVIDIA_DRIVER=1 2025-03-14T04:35:33.3086373Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2025-03-14T04:35:33.3086983Z + set -e 2025-03-14T04:35:33.3087244Z + '[' 1 -eq 0 ']' 2025-03-14T04:35:33.3087763Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2025-03-14T04:35:33.3088366Z + post_install_nvidia_driver_common 2025-03-14T04:35:33.3091328Z + sudo modprobe nvidia 2025-03-14T04:35:33.4645045Z + echo 'After installing NVIDIA driver' 2025-03-14T04:35:33.4645594Z + lspci 2025-03-14T04:35:33.4645891Z After installing NVIDIA driver 2025-03-14T04:35:33.4771632Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-03-14T04:35:33.4772324Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-03-14T04:35:33.4773047Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-03-14T04:35:33.4773728Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-03-14T04:35:33.4774350Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-03-14T04:35:33.4775040Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-03-14T04:35:33.4775699Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-03-14T04:35:33.4776297Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-03-14T04:35:33.4776889Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-03-14T04:35:33.4777480Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-03-14T04:35:33.4778101Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-03-14T04:35:33.4778842Z + lsmod 2025-03-14T04:35:33.4795522Z Module Size Used by 2025-03-14T04:35:33.4795975Z ib_core 454656 0 2025-03-14T04:35:33.4796327Z veth 36864 0 2025-03-14T04:35:33.4796678Z nvidia_modeset 1351680 0 2025-03-14T04:35:33.4797064Z video 65536 1 nvidia_modeset 2025-03-14T04:35:33.4797475Z wmi 36864 1 video 2025-03-14T04:35:33.4797848Z nvidia_uvm 4706304 0 2025-03-14T04:35:33.4798275Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2025-03-14T04:35:33.4798725Z drm 602112 1 nvidia 2025-03-14T04:35:33.4799138Z drm_panel_orientation_quirks 32768 1 drm 2025-03-14T04:35:33.4799620Z backlight 24576 3 video,drm,nvidia_modeset 2025-03-14T04:35:33.4800083Z i2c_core 110592 2 nvidia,drm 2025-03-14T04:35:33.4800479Z xt_conntrack 16384 1 2025-03-14T04:35:33.4800832Z nft_chain_nat 16384 3 2025-03-14T04:35:33.4801189Z xt_MASQUERADE 20480 1 2025-03-14T04:35:33.4801590Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-03-14T04:35:33.4802033Z nf_conntrack_netlink 57344 0 2025-03-14T04:35:33.4802561Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-03-14T04:35:33.4803146Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-03-14T04:35:33.4803560Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-03-14T04:35:33.4803961Z xfrm_user 57344 1 2025-03-14T04:35:33.4805069Z xfrm_algo 16384 1 xfrm_user 2025-03-14T04:35:33.4805596Z xt_addrtype 16384 2 2025-03-14T04:35:33.4805938Z nft_compat 20480 4 2025-03-14T04:35:33.4806339Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-03-14T04:35:33.4806877Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-03-14T04:35:33.4807367Z br_netfilter 36864 0 2025-03-14T04:35:33.4807730Z bridge 323584 1 br_netfilter 2025-03-14T04:35:33.4808125Z stp 16384 1 bridge 2025-03-14T04:35:33.4808604Z llc 16384 2 bridge,stp 2025-03-14T04:35:33.4808970Z overlay 167936 0 2025-03-14T04:35:33.4809293Z tls 135168 0 2025-03-14T04:35:33.4809614Z sunrpc 696320 1 2025-03-14T04:35:33.4809936Z nls_ascii 16384 1 2025-03-14T04:35:33.4810261Z nls_cp437 20480 1 2025-03-14T04:35:33.4810580Z vfat 24576 1 2025-03-14T04:35:33.4811059Z fat 86016 1 vfat 2025-03-14T04:35:33.4811411Z ena 180224 0 2025-03-14T04:35:33.4811735Z i8042 45056 0 2025-03-14T04:35:33.4812067Z serio 28672 3 i8042 2025-03-14T04:35:33.4812428Z skx_edac_common 28672 0 2025-03-14T04:35:33.4812766Z button 24576 0 2025-03-14T04:35:33.4813111Z ghash_clmulni_intel 16384 0 2025-03-14T04:35:33.4813461Z sch_fq_codel 20480 33 2025-03-14T04:35:33.4813807Z dm_mod 188416 0 2025-03-14T04:35:33.4814135Z fuse 163840 1 2025-03-14T04:35:33.4814467Z configfs 57344 1 2025-03-14T04:35:33.4814798Z loop 36864 0 2025-03-14T04:35:33.4815135Z dax 45056 1 dm_mod 2025-03-14T04:35:33.4815496Z dmi_sysfs 20480 0 2025-03-14T04:35:33.4815836Z crc32_pclmul 16384 0 2025-03-14T04:35:33.4816215Z crc32c_intel 24576 0 2025-03-14T04:35:33.4816537Z efivarfs 24576 1 2025-03-14T04:35:33.4816911Z + modinfo nvidia 2025-03-14T04:35:33.4817401Z filename: /lib/modules/6.1.128-136.201.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-03-14T04:35:33.4817990Z alias: char-major-195-* 2025-03-14T04:35:33.4818342Z version: 550.54.15 2025-03-14T04:35:33.4818671Z supported: external 2025-03-14T04:35:33.4818990Z license: NVIDIA 2025-03-14T04:35:33.4819337Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2025-03-14T04:35:33.4819860Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2025-03-14T04:35:33.4820298Z srcversion: 833721318DA517F0C2FEC97 2025-03-14T04:35:33.4820718Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-03-14T04:35:33.4821165Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-03-14T04:35:33.4821606Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-03-14T04:35:33.4822021Z depends: i2c-core,drm 2025-03-14T04:35:33.4822362Z retpoline: Y 2025-03-14T04:35:33.4822660Z name: nvidia 2025-03-14T04:35:33.4823139Z vermagic: 6.1.128-136.201.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-03-14T04:35:33.4823750Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-03-14T04:35:33.4824327Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-03-14T04:35:33.4824878Z parm: NVreg_ResmanDebugLevel:int 2025-03-14T04:35:33.4825288Z parm: NVreg_RmLogonRC:int 2025-03-14T04:35:33.4825678Z parm: NVreg_ModifyDeviceFiles:int 2025-03-14T04:35:33.4826097Z parm: NVreg_DeviceFileUID:int 2025-03-14T04:35:33.4826501Z parm: NVreg_DeviceFileGID:int 2025-03-14T04:35:33.4826910Z parm: NVreg_DeviceFileMode:int 2025-03-14T04:35:33.4827379Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-03-14T04:35:33.4827983Z parm: NVreg_UsePageAttributeTable:int 2025-03-14T04:35:33.4828412Z parm: NVreg_EnablePCIeGen3:int 2025-03-14T04:35:33.4828802Z parm: NVreg_EnableMSI:int 2025-03-14T04:35:33.4829254Z parm: NVreg_TCEBypassMode:int 2025-03-14T04:35:33.4829662Z parm: NVreg_EnableStreamMemOPs:int 2025-03-14T04:35:33.4830114Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-03-14T04:35:33.4830888Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-03-14T04:35:33.4831363Z parm: NVreg_EnableS0ixPowerManagement:int 2025-03-14T04:35:33.4831869Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-03-14T04:35:33.4832379Z parm: NVreg_DynamicPowerManagement:int 2025-03-14T04:35:33.4832988Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-03-14T04:35:33.4833687Z parm: NVreg_EnableGpuFirmware:int 2025-03-14T04:35:33.4834159Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-03-14T04:35:33.4834650Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-03-14T04:35:33.4835131Z parm: NVreg_EnableUserNUMAManagement:int 2025-03-14T04:35:33.4835591Z parm: NVreg_MemoryPoolSize:int 2025-03-14T04:35:33.4836029Z parm: NVreg_KMallocHeapMaxSize:int 2025-03-14T04:35:33.4836654Z parm: NVreg_VMallocHeapMaxSize:int 2025-03-14T04:35:33.4837090Z parm: NVreg_IgnoreMMIOCheck:int 2025-03-14T04:35:33.4837508Z parm: NVreg_NvLinkDisable:int 2025-03-14T04:35:33.4837965Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-03-14T04:35:33.4838442Z parm: NVreg_RegisterPCIDriver:int 2025-03-14T04:35:33.4838886Z parm: NVreg_EnableResizableBar:int 2025-03-14T04:35:33.4839337Z parm: NVreg_EnableDbgBreakpoint:int 2025-03-14T04:35:33.4839795Z parm: NVreg_EnableNonblockingOpen:int 2025-03-14T04:35:33.4840246Z parm: NVreg_RegistryDwords:charp 2025-03-14T04:35:33.4840703Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-03-14T04:35:33.4841149Z parm: NVreg_RmMsg:charp 2025-03-14T04:35:33.4841538Z parm: NVreg_GpuBlacklist:charp 2025-03-14T04:35:33.4841975Z parm: NVreg_TemporaryFilePath:charp 2025-03-14T04:35:33.4842417Z parm: NVreg_ExcludedGpus:charp 2025-03-14T04:35:33.4842844Z parm: NVreg_DmaRemapPeerMmio:int 2025-03-14T04:35:33.4843270Z parm: NVreg_RmNvlinkBandwidth:charp 2025-03-14T04:35:33.4843709Z parm: NVreg_ImexChannelCount:int 2025-03-14T04:35:33.4844132Z parm: rm_firmware_active:charp 2025-03-14T04:35:33.4844510Z + set +e 2025-03-14T04:35:33.4844776Z + nvidia-smi 2025-03-14T04:35:33.5327042Z Fri Mar 14 04:35:33 2025 2025-03-14T04:35:33.5327625Z +-----------------------------------------------------------------------------------------+ 2025-03-14T04:35:33.5328274Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2025-03-14T04:35:33.5328904Z |-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:35:33.5329543Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-03-14T04:35:33.5330257Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-03-14T04:35:33.5330821Z | | | MIG M. | 2025-03-14T04:35:33.5331255Z |=========================================+========================+======================| 2025-03-14T04:35:33.5515834Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2025-03-14T04:35:33.5516457Z | N/A 18C P8 10W / 70W | 0MiB / 15360MiB | 0% Default | 2025-03-14T04:35:33.5516976Z | | | N/A | 2025-03-14T04:35:33.5517504Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:35:33.5679581Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2025-03-14T04:35:33.5680436Z | N/A 19C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2025-03-14T04:35:33.5680957Z | | | N/A | 2025-03-14T04:35:33.5681493Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:35:33.5841117Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2025-03-14T04:35:33.5841718Z | N/A 19C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2025-03-14T04:35:33.5842255Z | | | N/A | 2025-03-14T04:35:33.5842793Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:35:33.6000831Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2025-03-14T04:35:33.6001435Z | N/A 18C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2025-03-14T04:35:33.6001967Z | | | N/A | 2025-03-14T04:35:33.6002501Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:35:33.6020378Z 2025-03-14T04:35:33.6020968Z +-----------------------------------------------------------------------------------------+ 2025-03-14T04:35:33.6021554Z | Processes: | 2025-03-14T04:35:33.6022147Z | GPU GI CI PID Type Process name GPU Memory | 2025-03-14T04:35:33.6022699Z | ID ID Usage | 2025-03-14T04:35:33.6023152Z |=========================================================================================| 2025-03-14T04:35:33.6049509Z | No running processes found | 2025-03-14T04:35:33.6050172Z +-----------------------------------------------------------------------------------------+ 2025-03-14T04:35:34.2229239Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-03-14T04:35:34.2416045Z Tesla T4 2025-03-14T04:35:34.2490202Z + NVIDIA_SMI_STATUS=0 2025-03-14T04:35:34.2490645Z + '[' 0 -eq 0 ']' 2025-03-14T04:35:34.2491357Z + echo 'INFO: Ignoring allowed status 0' 2025-03-14T04:35:34.2492079Z + set -e 2025-03-14T04:35:34.2492378Z INFO: Ignoring allowed status 0 2025-03-14T04:35:34.2496986Z == Installing nvidia container toolkit for amzn2023 == 2025-03-14T04:35:34.2501114Z + sudo yum install -y yum-utils 2025-03-14T04:35:34.7386575Z Last metadata expiration check: 3:11:09 ago on Fri Mar 14 01:24:25 2025. 2025-03-14T04:35:34.7637926Z Package dnf-utils-4.3.0-13.amzn2023.0.5.noarch is already installed. 2025-03-14T04:35:34.8079623Z Dependencies resolved. 2025-03-14T04:35:34.8317138Z Nothing to do. 2025-03-14T04:35:34.8317472Z Complete! 2025-03-14T04:35:34.8716673Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2025-03-14T04:35:34.8717456Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-03-14T04:35:34.8718571Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-03-14T04:35:35.1861740Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-03-14T04:35:35.2395109Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 2025-03-14T04:35:35.7997717Z nvidia-container-toolkit 16 kB/s | 833 B 00:00 2025-03-14T04:35:35.8258895Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2025-03-14T04:35:35.8263120Z Package nvidia-container-toolkit-1.16.2-1.x86_64 is already installed. 2025-03-14T04:35:35.8724734Z Dependencies resolved. 2025-03-14T04:35:35.8961550Z Nothing to do. 2025-03-14T04:35:35.8961898Z Complete! 2025-03-14T04:35:35.9363996Z + sudo systemctl restart docker 2025-03-14T04:37:42.3937547Z nvidia-persistenced failed to initialize. Check syslog for more details. 2025-03-14T04:37:42.4480416Z Fri Mar 14 04:37:42 2025 2025-03-14T04:37:42.4480955Z +-----------------------------------------------------------------------------------------+ 2025-03-14T04:37:42.4481624Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2025-03-14T04:37:42.4482314Z |-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:37:42.4482991Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-03-14T04:37:42.4483708Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-03-14T04:37:42.4484292Z | | | MIG M. | 2025-03-14T04:37:42.4484909Z |=========================================+========================+======================| 2025-03-14T04:37:42.4664006Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2025-03-14T04:37:42.4664605Z | N/A 18C P8 8W / 70W | 0MiB / 15360MiB | 0% Default | 2025-03-14T04:37:42.4667048Z | | | N/A | 2025-03-14T04:37:42.4668551Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:37:42.4826042Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2025-03-14T04:37:42.4826656Z | N/A 19C P8 10W / 70W | 0MiB / 15360MiB | 0% Default | 2025-03-14T04:37:42.4827204Z | | | N/A | 2025-03-14T04:37:42.4827723Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:37:42.4983360Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2025-03-14T04:37:42.4983965Z | N/A 19C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2025-03-14T04:37:42.4984466Z | | | N/A | 2025-03-14T04:37:42.4984988Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:37:42.5142302Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2025-03-14T04:37:42.5142948Z | N/A 18C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2025-03-14T04:37:42.5143451Z | | | N/A | 2025-03-14T04:37:42.5143970Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T04:37:42.5157378Z 2025-03-14T04:37:42.5157927Z +-----------------------------------------------------------------------------------------+ 2025-03-14T04:37:42.5158501Z | Processes: | 2025-03-14T04:37:42.5159100Z | GPU GI CI PID Type Process name GPU Memory | 2025-03-14T04:37:42.5159680Z | ID ID Usage | 2025-03-14T04:37:42.5160138Z |=========================================================================================| 2025-03-14T04:37:42.5187612Z | No running processes found | 2025-03-14T04:37:42.5188306Z +-----------------------------------------------------------------------------------------+ 2025-03-14T04:37:43.9110604Z Command completed after 1 attempt(s). 2025-03-14T04:37:43.9215784Z Prepare all required actions 2025-03-14T04:37:43.9249068Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-03-14T04:37:43.9249458Z with: 2025-03-14T04:37:43.9249915Z github-token: *** 2025-03-14T04:37:43.9250180Z env: 2025-03-14T04:37:43.9250429Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:37:43.9250810Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:37:43.9251234Z ##[endgroup] 2025-03-14T04:37:43.9273998Z ##[group]Run set -eux 2025-03-14T04:37:43.9274359Z set -eux 2025-03-14T04:37:43.9274905Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-03-14T04:37:43.9284947Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:37:43.9285506Z env: 2025-03-14T04:37:43.9285769Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:37:43.9286168Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:37:43.9286760Z GITHUB_TOKEN: *** 2025-03-14T04:37:43.9287044Z ##[endgroup] 2025-03-14T04:37:43.9315655Z + python3 .github/scripts/get_workflow_job_id.py 13849515290 i-0f8cc103977fc9a05 2025-03-14T04:37:45.8844329Z setting job-id=38755323799 2025-03-14T04:37:45.8845123Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T04:37:45.8993985Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 dataclasses_json==0.6.7 2025-03-14T04:37:45.8994922Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 dataclasses_json==0.6.7 2025-03-14T04:37:45.8995687Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2025-03-14T04:37:45.8996296Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-03-14T04:37:45.9003881Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:37:45.9004367Z env: 2025-03-14T04:37:45.9004661Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:37:45.9005093Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:37:45.9005683Z JOB_ID: 38755323799 2025-03-14T04:37:45.9006246Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T04:37:45.9006870Z WORKFLOW_NAME: pull 2025-03-14T04:37:45.9007180Z WORKFLOW_RUN_ID: 13849515290 2025-03-14T04:37:45.9007552Z ##[endgroup] 2025-03-14T04:37:46.1531368Z Defaulting to user installation because normal site-packages is not writeable 2025-03-14T04:37:46.1709287Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2025-03-14T04:37:46.1712735Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2025-03-14T04:37:46.1717932Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-03-14T04:37:46.1829193Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-03-14T04:37:46.1833817Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-03-14T04:37:46.1928283Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (24.2) 2025-03-14T04:37:46.1959902Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.12.2) 2025-03-14T04:37:46.1964114Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.0.0) 2025-03-14T04:37:46.3948946Z Prepare all required actions 2025-03-14T04:37:46.3949458Z Getting action download info 2025-03-14T04:37:46.5105533Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-03-14T04:37:46.7041891Z Download action repository 'actions/download-artifact@v4' (SHA:cc203385981b70ca67e1cc392babf9cc229d5806) 2025-03-14T04:37:46.9443297Z ##[group]Run ./.github/actions/download-build-artifacts 2025-03-14T04:37:46.9443758Z with: 2025-03-14T04:37:46.9444066Z name: linux-focal-cuda11.8-py3.10-gcc9 2025-03-14T04:37:46.9444479Z s3-bucket: gha-artifacts 2025-03-14T04:37:46.9444963Z env: 2025-03-14T04:37:46.9445229Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:37:46.9445640Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:37:46.9446073Z ##[endgroup] 2025-03-14T04:37:46.9499606Z ##[group]Run seemethere/download-artifact-s3@v4 2025-03-14T04:37:46.9500023Z with: 2025-03-14T04:37:46.9500381Z name: linux-focal-cuda11.8-py3.10-gcc9 2025-03-14T04:37:46.9500766Z s3-bucket: gha-artifacts 2025-03-14T04:37:46.9501093Z region: us-east-1 2025-03-14T04:37:46.9501376Z env: 2025-03-14T04:37:46.9501642Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:37:46.9502044Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:37:46.9502479Z ##[endgroup] 2025-03-14T04:37:47.4475736Z (node:750408) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-03-14T04:37:47.4476360Z 2025-03-14T04:37:47.4476609Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-03-14T04:37:47.4477275Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-03-14T04:37:47.4477999Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-03-14T04:37:47.5665755Z Found 1 objects with prefix pytorch/pytorch/13849515290/linux-focal-cuda11.8-py3.10-gcc9/ 2025-03-14T04:37:47.5666667Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-03-14T04:38:06.6103681Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-03-14T04:38:06.6109770Z Artifact download has finished successfully 2025-03-14T04:38:06.6346042Z ##[group]Run unzip -o artifacts.zip 2025-03-14T04:38:06.6346443Z unzip -o artifacts.zip 2025-03-14T04:38:06.6352555Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:38:06.6353113Z env: 2025-03-14T04:38:06.6353542Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:06.6353965Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:06.6354416Z ##[endgroup] 2025-03-14T04:38:06.6398098Z Archive: artifacts.zip 2025-03-14T04:38:06.6399057Z creating: dist/ 2025-03-14T04:38:08.8394874Z inflating: dist/torch-2.8.0a0+gitaed0b7a-cp310-cp310-linux_x86_64.whl 2025-03-14T04:38:08.8527789Z inflating: dist/.ninja_log 2025-03-14T04:38:08.8528543Z creating: build/custom_test_artifacts/ 2025-03-14T04:38:08.8531320Z creating: build/custom_test_artifacts/custom-op-build/ 2025-03-14T04:38:08.8531957Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-03-14T04:38:08.8532665Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-03-14T04:38:08.8535948Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-03-14T04:38:08.8536746Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/ 2025-03-14T04:38:08.8537536Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeSystem.cmake 2025-03-14T04:38:08.8538380Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/ 2025-03-14T04:38:08.8539211Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/tmp/ 2025-03-14T04:38:08.8540148Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c 2025-03-14T04:38:08.8541589Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/a.out 2025-03-14T04:38:08.8542503Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake 2025-03-14T04:38:08.8543378Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/ 2025-03-14T04:38:08.8544220Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/ 2025-03-14T04:38:08.8545201Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-03-14T04:38:08.8546537Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out 2025-03-14T04:38:08.8547481Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake 2025-03-14T04:38:08.8549041Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin 2025-03-14T04:38:08.8550854Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin 2025-03-14T04:38:08.8551812Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/ 2025-03-14T04:38:08.8552667Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/ 2025-03-14T04:38:08.8600475Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-03-14T04:38:08.8601768Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-03-14T04:38:08.8603051Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-03-14T04:38:08.8604379Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-03-14T04:38:08.8605770Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-03-14T04:38:08.8606967Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-03-14T04:38:08.8608165Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-03-14T04:38:08.8609375Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-03-14T04:38:08.8610568Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-03-14T04:38:08.8648681Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-03-14T04:38:08.8691354Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-03-14T04:38:08.8692608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-03-14T04:38:08.8693733Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-03-14T04:38:08.8694819Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-03-14T04:38:08.8695871Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-03-14T04:38:08.8696923Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-03-14T04:38:08.8697964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o 2025-03-14T04:38:08.8699015Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-03-14T04:38:08.8761326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out 2025-03-14T04:38:08.8762339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake 2025-03-14T04:38:08.8826341Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin 2025-03-14T04:38:08.8827313Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-03-14T04:38:08.8829102Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2025-03-14T04:38:08.8829874Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-03-14T04:38:08.8830697Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-03-14T04:38:08.8831575Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-03-14T04:38:08.8832580Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-03-14T04:38:08.8833839Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-03-14T04:38:08.8834759Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-03-14T04:38:08.8835704Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-03-14T04:38:08.8836661Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-03-14T04:38:08.8837611Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-03-14T04:38:08.8838569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-03-14T04:38:08.8839509Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-03-14T04:38:08.8854200Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-03-14T04:38:08.9055541Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-03-14T04:38:08.9056449Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-03-14T04:38:08.9057393Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-03-14T04:38:08.9058444Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-03-14T04:38:08.9059449Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-03-14T04:38:08.9060576Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-03-14T04:38:08.9061553Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-03-14T04:38:08.9062545Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-03-14T04:38:08.9063521Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-03-14T04:38:08.9064496Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-03-14T04:38:08.9065457Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-03-14T04:38:08.9080902Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-03-14T04:38:08.9168432Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-03-14T04:38:08.9169701Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-03-14T04:38:08.9170644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-03-14T04:38:08.9171684Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-03-14T04:38:08.9172478Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-03-14T04:38:08.9173260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-03-14T04:38:08.9174033Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2025-03-14T04:38:08.9174757Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-03-14T04:38:08.9175416Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-03-14T04:38:08.9176090Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-03-14T04:38:08.9342478Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-03-14T04:38:08.9409638Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-03-14T04:38:08.9410485Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-03-14T04:38:08.9411069Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-03-14T04:38:08.9411757Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-03-14T04:38:08.9416852Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-03-14T04:38:08.9417643Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/ 2025-03-14T04:38:08.9418414Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeSystem.cmake 2025-03-14T04:38:08.9419249Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/ 2025-03-14T04:38:08.9420056Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/tmp/ 2025-03-14T04:38:08.9420992Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c 2025-03-14T04:38:08.9421964Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/a.out 2025-03-14T04:38:08.9423183Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake 2025-03-14T04:38:08.9424040Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/ 2025-03-14T04:38:08.9424870Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/ 2025-03-14T04:38:08.9426206Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-03-14T04:38:08.9427628Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out 2025-03-14T04:38:08.9428543Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake 2025-03-14T04:38:08.9430115Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin 2025-03-14T04:38:08.9431911Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin 2025-03-14T04:38:08.9432961Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/ 2025-03-14T04:38:08.9433988Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/ 2025-03-14T04:38:08.9481219Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-03-14T04:38:08.9482509Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-03-14T04:38:08.9483774Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-03-14T04:38:08.9485194Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-03-14T04:38:08.9486649Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-03-14T04:38:08.9487852Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-03-14T04:38:08.9489036Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-03-14T04:38:08.9490233Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-03-14T04:38:08.9491415Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-03-14T04:38:08.9528444Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-03-14T04:38:08.9570962Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-03-14T04:38:08.9572197Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-03-14T04:38:08.9573305Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-03-14T04:38:08.9574368Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-03-14T04:38:08.9575419Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-03-14T04:38:08.9576476Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-03-14T04:38:08.9577499Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o 2025-03-14T04:38:08.9578546Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-03-14T04:38:08.9641294Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out 2025-03-14T04:38:08.9642278Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake 2025-03-14T04:38:08.9706530Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin 2025-03-14T04:38:08.9707487Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-03-14T04:38:08.9708218Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2025-03-14T04:38:08.9708973Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-03-14T04:38:08.9709774Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-03-14T04:38:08.9710885Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-03-14T04:38:08.9711913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-03-14T04:38:08.9713008Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-03-14T04:38:08.9714122Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-03-14T04:38:08.9715103Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-03-14T04:38:08.9716090Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-03-14T04:38:08.9717072Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-03-14T04:38:08.9718059Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-03-14T04:38:08.9719029Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-03-14T04:38:08.9734312Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-03-14T04:38:08.9799736Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-03-14T04:38:08.9800788Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-03-14T04:38:08.9801735Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-03-14T04:38:08.9802596Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-03-14T04:38:08.9803387Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-03-14T04:38:08.9804157Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-03-14T04:38:08.9804946Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2025-03-14T04:38:08.9806293Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-03-14T04:38:08.9807091Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-03-14T04:38:08.9807761Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-03-14T04:38:08.9860147Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-03-14T04:38:08.9860826Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-03-14T04:38:08.9861460Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-03-14T04:38:08.9862203Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-03-14T04:38:08.9867754Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-03-14T04:38:08.9868603Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/ 2025-03-14T04:38:08.9869757Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeSystem.cmake 2025-03-14T04:38:08.9870674Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/ 2025-03-14T04:38:08.9871562Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/tmp/ 2025-03-14T04:38:08.9872554Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c 2025-03-14T04:38:08.9873828Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/a.out 2025-03-14T04:38:08.9874819Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake 2025-03-14T04:38:08.9875779Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/ 2025-03-14T04:38:08.9876710Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/ 2025-03-14T04:38:08.9877968Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-03-14T04:38:08.9879075Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out 2025-03-14T04:38:08.9880089Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake 2025-03-14T04:38:08.9881421Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin 2025-03-14T04:38:08.9883175Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin 2025-03-14T04:38:08.9884219Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/ 2025-03-14T04:38:08.9885256Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/ 2025-03-14T04:38:08.9932989Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-03-14T04:38:08.9934508Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-03-14T04:38:08.9935818Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-03-14T04:38:08.9937151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-03-14T04:38:08.9938474Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-03-14T04:38:08.9939739Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-03-14T04:38:08.9941007Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-03-14T04:38:08.9942284Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-03-14T04:38:08.9943557Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-03-14T04:38:08.9978414Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-03-14T04:38:09.0020784Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-03-14T04:38:09.0022097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-03-14T04:38:09.0023289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-03-14T04:38:09.0024440Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-03-14T04:38:09.0025559Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-03-14T04:38:09.0026695Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-03-14T04:38:09.0027795Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o 2025-03-14T04:38:09.0028911Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-03-14T04:38:09.0091992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out 2025-03-14T04:38:09.0093028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake 2025-03-14T04:38:09.0154781Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin 2025-03-14T04:38:09.0156071Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-03-14T04:38:09.0156891Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2025-03-14T04:38:09.0157735Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-03-14T04:38:09.0158621Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-03-14T04:38:09.0159622Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-03-14T04:38:09.0160769Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-03-14T04:38:09.0161862Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-03-14T04:38:09.0162887Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-03-14T04:38:09.0163946Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-03-14T04:38:09.0165133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-03-14T04:38:09.0166300Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-03-14T04:38:09.0167342Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-03-14T04:38:09.0168361Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-03-14T04:38:09.0169637Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-03-14T04:38:09.0291488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-03-14T04:38:09.0292577Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-03-14T04:38:09.0293623Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-03-14T04:38:09.0294784Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-03-14T04:38:09.0295896Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-03-14T04:38:09.0296937Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-03-14T04:38:09.0298009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-03-14T04:38:09.0299106Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-03-14T04:38:09.0300197Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-03-14T04:38:09.0301285Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-03-14T04:38:09.0302357Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-03-14T04:38:09.0316319Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-03-14T04:38:09.0374446Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-03-14T04:38:09.0375622Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-03-14T04:38:09.0376615Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-03-14T04:38:09.0377715Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-03-14T04:38:09.0378542Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-03-14T04:38:09.0379352Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-03-14T04:38:09.0380157Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2025-03-14T04:38:09.0380915Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-03-14T04:38:09.0381615Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-03-14T04:38:09.0382335Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-03-14T04:38:09.0488607Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-03-14T04:38:09.0532869Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-03-14T04:38:09.0533668Z creating: build/lib/ 2025-03-14T04:38:09.0626337Z inflating: build/lib/libprotobuf-lite.a 2025-03-14T04:38:09.1091986Z inflating: build/lib/libprotobuf.a 2025-03-14T04:38:09.1610434Z inflating: build/lib/libprotoc.a 2025-03-14T04:38:09.1619520Z inflating: build/lib/libpthreadpool.a 2025-03-14T04:38:09.1627879Z inflating: build/lib/libcpuinfo.a 2025-03-14T04:38:09.1636466Z inflating: build/lib/libcpuinfo_internals.a 2025-03-14T04:38:09.1637407Z inflating: build/lib/libclog.a 2025-03-14T04:38:09.1659865Z inflating: build/lib/libpytorch_qnnpack.a 2025-03-14T04:38:09.1660755Z inflating: build/lib/libnnpack_reference_layers.a 2025-03-14T04:38:09.1680798Z inflating: build/lib/libnnpack.a 2025-03-14T04:38:09.1870763Z inflating: build/lib/libmicrokernels-prod.a 2025-03-14T04:38:09.2700203Z inflating: build/lib/libmicrokernels-all.a 2025-03-14T04:38:09.2770953Z inflating: build/lib/libgtest.a 2025-03-14T04:38:09.2789919Z inflating: build/lib/libgmock.a 2025-03-14T04:38:09.2790778Z inflating: build/lib/libgtest_main.a 2025-03-14T04:38:09.2791248Z inflating: build/lib/libgmock_main.a 2025-03-14T04:38:09.2886136Z inflating: build/lib/libXNNPACK.a 2025-03-14T04:38:09.2961776Z inflating: build/lib/libbenchmark.a 2025-03-14T04:38:09.2962262Z inflating: build/lib/libbenchmark_main.a 2025-03-14T04:38:09.3030088Z inflating: build/lib/libasmjit.a 2025-03-14T04:38:09.3036799Z inflating: build/lib/libittnotify.a 2025-03-14T04:38:09.4301618Z inflating: build/lib/libfbgemm.a 2025-03-14T04:38:09.4331721Z inflating: build/lib/libtensorpipe_uv.a 2025-03-14T04:38:09.4911451Z inflating: build/lib/libtensorpipe.a 2025-03-14T04:38:09.5172647Z inflating: build/lib/libtensorpipe_cuda.a 2025-03-14T04:38:09.5305450Z inflating: build/lib/libgloo.a 2025-03-14T04:38:09.5349051Z inflating: build/lib/libonnx_proto.a 2025-03-14T04:38:09.6098005Z inflating: build/lib/libonnx.a 2025-03-14T04:38:10.6734511Z inflating: build/lib/libdnnl.a 2025-03-14T04:38:10.6754759Z inflating: build/lib/libfmt.a 2025-03-14T04:38:10.7236093Z inflating: build/lib/libkineto.a 2025-03-14T04:38:10.7354269Z inflating: build/lib/libc10.so 2025-03-14T04:38:10.7417962Z inflating: build/lib/libc10_cuda.so 2025-03-14T04:38:10.7418419Z inflating: build/lib/libcaffe2_nvrtc.so 2025-03-14T04:38:10.7419090Z inflating: build/lib/libtorch_global_deps.so 2025-03-14T04:38:13.3485282Z inflating: build/lib/libtorch_cpu.so 2025-03-14T04:38:13.3489952Z inflating: build/lib/libunbox_lib.a 2025-03-14T04:38:13.3494241Z inflating: build/lib/libshm.so 2025-03-14T04:38:13.3879204Z inflating: build/lib/libgloo_cuda.a 2025-03-14T04:38:15.4769066Z inflating: build/lib/libtorch_cuda.so 2025-03-14T04:38:15.4769758Z inflating: build/lib/libtorch.so 2025-03-14T04:38:16.4320687Z inflating: build/lib/libtorch_cuda_linalg.so 2025-03-14T04:38:16.4394320Z inflating: build/lib/libtorchbind_test.so 2025-03-14T04:38:16.4415755Z inflating: build/lib/libjitbackend_test.so 2025-03-14T04:38:16.4441165Z inflating: build/lib/libbackend_with_compiler.so 2025-03-14T04:38:16.4467995Z inflating: build/lib/libaoti_custom_ops.so 2025-03-14T04:38:16.4468954Z inflating: build/lib/libc10d_cuda_test.so 2025-03-14T04:38:16.6587527Z inflating: build/lib/libtorch_python.so 2025-03-14T04:38:16.6623758Z inflating: build/lib/libnnapi_backend.so 2025-03-14T04:38:16.6624447Z creating: build/bin/ 2025-03-14T04:38:16.7089369Z inflating: build/bin/protoc-3.13.0.0 2025-03-14T04:38:16.7554533Z inflating: build/bin/protoc 2025-03-14T04:38:16.7612433Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-03-14T04:38:16.7669554Z inflating: build/bin/c10_DeviceGuard_test 2025-03-14T04:38:16.7728117Z inflating: build/bin/c10_Device_test 2025-03-14T04:38:16.7794641Z inflating: build/bin/c10_DispatchKeySet_test 2025-03-14T04:38:16.7849759Z inflating: build/bin/c10_StreamGuard_test 2025-03-14T04:38:16.7909573Z inflating: build/bin/c10_Scalar_test 2025-03-14T04:38:16.7967657Z inflating: build/bin/c10_SymInt_test 2025-03-14T04:38:16.8026763Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-03-14T04:38:16.8090209Z inflating: build/bin/c10_InlineStreamGuard_test 2025-03-14T04:38:16.8151675Z inflating: build/bin/c10_SizesAndStrides_test 2025-03-14T04:38:16.8230035Z inflating: build/bin/c10_cow_test 2025-03-14T04:38:16.8283994Z inflating: build/bin/c10_ConstexprCrc_test 2025-03-14T04:38:16.8340263Z inflating: build/bin/c10_DeadlockDetection_test 2025-03-14T04:38:16.8397012Z inflating: build/bin/c10_ArrayRef_test 2025-03-14T04:38:16.8458046Z inflating: build/bin/c10_Bitset_test 2025-03-14T04:38:16.8515225Z inflating: build/bin/c10_Half_test 2025-03-14T04:38:16.8571327Z inflating: build/bin/c10_Synchronized_test 2025-03-14T04:38:16.8634030Z inflating: build/bin/c10_LeftRight_test 2025-03-14T04:38:16.8695856Z inflating: build/bin/c10_Metaprogramming_test 2025-03-14T04:38:16.8754225Z inflating: build/bin/c10_NetworkFlow_test 2025-03-14T04:38:16.8818006Z inflating: build/bin/c10_ThreadLocal_test 2025-03-14T04:38:16.8874313Z inflating: build/bin/c10_TypeList_test 2025-03-14T04:38:16.8929581Z inflating: build/bin/c10_TypeTraits_test 2025-03-14T04:38:16.8988710Z inflating: build/bin/c10_TypeIndex_test 2025-03-14T04:38:16.9045100Z inflating: build/bin/c10_accumulate_test 2025-03-14T04:38:16.9108383Z inflating: build/bin/c10_bfloat16_test 2025-03-14T04:38:16.9163439Z inflating: build/bin/c10_bit_cast_test 2025-03-14T04:38:16.9226497Z inflating: build/bin/c10_complex_test 2025-03-14T04:38:16.9282052Z inflating: build/bin/c10_error_test 2025-03-14T04:38:16.9346492Z inflating: build/bin/c10_complex_math_test 2025-03-14T04:38:16.9407720Z inflating: build/bin/c10_exception_test 2025-03-14T04:38:16.9464537Z inflating: build/bin/c10_flags_test 2025-03-14T04:38:16.9519722Z inflating: build/bin/c10_generic_math_test 2025-03-14T04:38:16.9577940Z inflating: build/bin/c10_irange_test 2025-03-14T04:38:16.9754334Z inflating: build/bin/c10_intrusive_ptr_test 2025-03-14T04:38:16.9816141Z inflating: build/bin/c10_lazy_test 2025-03-14T04:38:16.9878541Z inflating: build/bin/c10_logging_test 2025-03-14T04:38:16.9959653Z inflating: build/bin/c10_optional_test 2025-03-14T04:38:17.0029934Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-03-14T04:38:17.0090726Z inflating: build/bin/c10_registry_test 2025-03-14T04:38:17.0147828Z inflating: build/bin/c10_ssize_test 2025-03-14T04:38:17.0312766Z inflating: build/bin/c10_small_vector_test 2025-03-14T04:38:17.0371142Z inflating: build/bin/c10_string_util_test 2025-03-14T04:38:17.0425602Z inflating: build/bin/c10_string_view_test 2025-03-14T04:38:17.0481195Z inflating: build/bin/c10_tempfile_test 2025-03-14T04:38:17.0533280Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-03-14T04:38:17.0595418Z inflating: build/bin/c10_typeid_test 2025-03-14T04:38:17.0655147Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2025-03-14T04:38:17.0713495Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2025-03-14T04:38:17.0771141Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2025-03-14T04:38:17.0830612Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2025-03-14T04:38:17.0886873Z inflating: build/bin/c10_cuda_CUDATest 2025-03-14T04:38:17.0944767Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2025-03-14T04:38:17.1003802Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2025-03-14T04:38:17.1063086Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2025-03-14T04:38:17.1481053Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-03-14T04:38:17.1908838Z inflating: build/bin/vec_test_all_types_AVX512 2025-03-14T04:38:17.2349544Z inflating: build/bin/vec_test_all_types_AVX2 2025-03-14T04:38:17.2410828Z inflating: build/bin/FileStoreTest 2025-03-14T04:38:17.2469641Z inflating: build/bin/HashStoreTest 2025-03-14T04:38:17.2528681Z inflating: build/bin/BackoffTest 2025-03-14T04:38:17.2590414Z inflating: build/bin/TCPStoreTest 2025-03-14T04:38:17.2650090Z inflating: build/bin/test_edge_op_registration 2025-03-14T04:38:17.2665149Z inflating: build/bin/ProcessGroupMPITest 2025-03-14T04:38:17.2667995Z inflating: build/bin/torch_shm_manager 2025-03-14T04:38:17.2750012Z inflating: build/bin/Dict_test 2025-03-14T04:38:17.2809742Z inflating: build/bin/Dimname_test 2025-03-14T04:38:17.2880939Z inflating: build/bin/MaybeOwned_test 2025-03-14T04:38:17.2945894Z inflating: build/bin/NamedTensor_test 2025-03-14T04:38:17.3012336Z inflating: build/bin/apply_utils_test 2025-03-14T04:38:17.3076831Z inflating: build/bin/atest 2025-03-14T04:38:17.3146745Z inflating: build/bin/basic 2025-03-14T04:38:17.3209988Z inflating: build/bin/broadcast_test 2025-03-14T04:38:17.3266365Z inflating: build/bin/cpu_allocator_test 2025-03-14T04:38:17.3332401Z inflating: build/bin/cpu_generator_test 2025-03-14T04:38:17.3393307Z inflating: build/bin/cpu_profiling_allocator_test 2025-03-14T04:38:17.3492775Z inflating: build/bin/cpu_rng_test 2025-03-14T04:38:17.3548528Z inflating: build/bin/dispatch_key_set_test 2025-03-14T04:38:17.3605137Z inflating: build/bin/dlconvertor_test 2025-03-14T04:38:17.3670127Z inflating: build/bin/extension_backend_test 2025-03-14T04:38:17.3731546Z inflating: build/bin/half_test 2025-03-14T04:38:17.3837801Z inflating: build/bin/ivalue_test 2025-03-14T04:38:17.3895514Z inflating: build/bin/lazy_tensor_test 2025-03-14T04:38:17.3954413Z inflating: build/bin/math_kernel_test 2025-03-14T04:38:17.4016531Z inflating: build/bin/memory_format_test 2025-03-14T04:38:17.4074238Z inflating: build/bin/memory_overlapping_test 2025-03-14T04:38:17.4133883Z inflating: build/bin/mobile_memory_cleanup 2025-03-14T04:38:17.4196634Z inflating: build/bin/native_test 2025-03-14T04:38:17.4255776Z inflating: build/bin/operator_name_test 2025-03-14T04:38:17.4313251Z inflating: build/bin/operators_test 2025-03-14T04:38:17.4372634Z inflating: build/bin/packedtensoraccessor_test 2025-03-14T04:38:17.4444520Z inflating: build/bin/pow_test 2025-03-14T04:38:17.4509411Z inflating: build/bin/quantized_test 2025-03-14T04:38:17.4564991Z inflating: build/bin/reduce_ops_test 2025-03-14T04:38:17.4623179Z inflating: build/bin/reportMemoryUsage_test 2025-03-14T04:38:17.4687806Z inflating: build/bin/scalar_tensor_test 2025-03-14T04:38:17.4751959Z inflating: build/bin/scalar_test 2025-03-14T04:38:17.4811473Z inflating: build/bin/StorageUtils_test 2025-03-14T04:38:17.4870454Z inflating: build/bin/stride_properties_test 2025-03-14T04:38:17.4954527Z inflating: build/bin/tensor_iterator_test 2025-03-14T04:38:17.5017234Z inflating: build/bin/test_parallel 2025-03-14T04:38:17.5018510Z inflating: build/bin/thread_init_test 2025-03-14T04:38:17.5080899Z inflating: build/bin/type_ptr_test 2025-03-14T04:38:17.5148706Z inflating: build/bin/type_test 2025-03-14T04:38:17.5209280Z inflating: build/bin/undefined_tensor_test 2025-03-14T04:38:17.5210024Z inflating: build/bin/verify_api_visibility 2025-03-14T04:38:17.5290581Z inflating: build/bin/legacy_vmap_test 2025-03-14T04:38:17.5349452Z inflating: build/bin/weakref_test 2025-03-14T04:38:17.5408819Z inflating: build/bin/wrapdim_test 2025-03-14T04:38:17.5466060Z inflating: build/bin/xla_tensor_test 2025-03-14T04:38:17.5532218Z inflating: build/bin/IListRef_test 2025-03-14T04:38:17.5648778Z inflating: build/bin/List_test 2025-03-14T04:38:17.5720771Z inflating: build/bin/KernelFunction_test 2025-03-14T04:38:17.5852829Z inflating: build/bin/kernel_function_legacy_test 2025-03-14T04:38:17.5955450Z inflating: build/bin/kernel_function_test 2025-03-14T04:38:17.6093918Z inflating: build/bin/kernel_lambda_legacy_test 2025-03-14T04:38:17.6204000Z inflating: build/bin/kernel_lambda_test 2025-03-14T04:38:17.6271354Z inflating: build/bin/kernel_stackbased_test 2025-03-14T04:38:17.6377408Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-03-14T04:38:17.6434072Z inflating: build/bin/CppSignature_test 2025-03-14T04:38:17.6496880Z inflating: build/bin/backend_fallback_test 2025-03-14T04:38:17.6550820Z inflating: build/bin/op_allowlist_test 2025-03-14T04:38:17.6864549Z inflating: build/bin/op_registration_test 2025-03-14T04:38:17.6935246Z inflating: build/bin/inline_container_test 2025-03-14T04:38:17.6994527Z inflating: build/bin/cuda_allocator_test 2025-03-14T04:38:17.7053957Z inflating: build/bin/cuda_apply_test 2025-03-14T04:38:17.7118903Z inflating: build/bin/cuda_atomic_ops_test 2025-03-14T04:38:17.7184460Z inflating: build/bin/cuda_caching_host_allocator_test 2025-03-14T04:38:17.7261231Z inflating: build/bin/cuda_complex_math_test 2025-03-14T04:38:17.7327958Z inflating: build/bin/cuda_complex_test 2025-03-14T04:38:17.7394254Z inflating: build/bin/cuda_cub_test 2025-03-14T04:38:17.7449831Z inflating: build/bin/cuda_device_test 2025-03-14T04:38:17.7520591Z inflating: build/bin/cuda_distributions_test 2025-03-14T04:38:17.7578823Z inflating: build/bin/cuda_dlconvertor_test 2025-03-14T04:38:17.7636108Z inflating: build/bin/cuda_reportMemoryUsage_test 2025-03-14T04:38:17.7693676Z inflating: build/bin/cuda_allocatorTraceTracker_test 2025-03-14T04:38:17.7749925Z inflating: build/bin/cuda_integer_divider_test 2025-03-14T04:38:17.7819543Z inflating: build/bin/cuda_stream_test 2025-03-14T04:38:17.7875908Z inflating: build/bin/cuda_cudnn_test 2025-03-14T04:38:17.7931941Z inflating: build/bin/cuda_optional_test 2025-03-14T04:38:17.7989000Z inflating: build/bin/cuda_half_test 2025-03-14T04:38:17.8052122Z inflating: build/bin/cuda_generator_test 2025-03-14T04:38:17.8109238Z inflating: build/bin/cuda_packedtensoraccessor_test 2025-03-14T04:38:17.8168203Z inflating: build/bin/cuda_vectorized_test 2025-03-14T04:38:17.8771890Z inflating: build/bin/test_jit 2025-03-14T04:38:17.9634570Z inflating: build/bin/test_tensorexpr 2025-03-14T04:38:17.9650132Z inflating: build/bin/tutorial_tensorexpr 2025-03-14T04:38:17.9651780Z inflating: build/bin/example_allreduce 2025-03-14T04:38:17.9714310Z inflating: build/bin/test_dist_autograd 2025-03-14T04:38:17.9788450Z inflating: build/bin/ProcessGroupGlooTest 2025-03-14T04:38:17.9852785Z inflating: build/bin/ProcessGroupGlooAsyncTest 2025-03-14T04:38:17.9910977Z inflating: build/bin/ProcessGroupUCCTest 2025-03-14T04:38:17.9988864Z inflating: build/bin/test_cpp_rpc 2025-03-14T04:38:18.0057660Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2025-03-14T04:38:18.0129329Z inflating: build/bin/ProcessGroupNCCLTest 2025-03-14T04:38:18.0130505Z inflating: build/bin/parallel_benchmark 2025-03-14T04:38:18.1369807Z inflating: build/bin/test_api 2025-03-14T04:38:18.1441843Z inflating: build/bin/test_mobile_nnc 2025-03-14T04:38:18.1451345Z inflating: build/bin/aot_model_compiler_test 2025-03-14T04:38:18.1817737Z inflating: build/bin/test_lazy 2025-03-14T04:38:18.1818403Z creating: .additional_ci_files/ 2025-03-14T04:38:18.1930570Z inflating: .additional_ci_files/test-times.json 2025-03-14T04:38:18.2360867Z inflating: .additional_ci_files/test-class-times.json 2025-03-14T04:38:18.2398696Z ##[group]Run rm artifacts.zip 2025-03-14T04:38:18.2399083Z rm artifacts.zip 2025-03-14T04:38:18.2406927Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:38:18.2407353Z env: 2025-03-14T04:38:18.2407591Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:18.2407964Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:18.2408370Z ##[endgroup] 2025-03-14T04:38:18.3064756Z ##[group]Run df -H 2025-03-14T04:38:18.3065051Z df -H 2025-03-14T04:38:18.3071181Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:38:18.3071638Z env: 2025-03-14T04:38:18.3071907Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:18.3072449Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:18.3072965Z ##[endgroup] 2025-03-14T04:38:18.3114983Z Filesystem Size Used Avail Use% Mounted on 2025-03-14T04:38:18.3115482Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-03-14T04:38:18.3115908Z tmpfs 101G 8.2M 101G 1% /dev/shm 2025-03-14T04:38:18.3116332Z tmpfs 41G 697k 41G 1% /run 2025-03-14T04:38:18.3116742Z /dev/nvme0n1p1 161G 60G 102G 37% / 2025-03-14T04:38:18.3117186Z tmpfs 101G 33k 101G 1% /tmp 2025-03-14T04:38:18.3117633Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-03-14T04:38:18.3118093Z tmpfs 21G 0 21G 0% /run/user/0 2025-03-14T04:38:18.3152451Z Prepare all required actions 2025-03-14T04:38:18.3153002Z Getting action download info 2025-03-14T04:38:18.4408918Z ##[group]Run ./.github/actions/download-td-artifacts 2025-03-14T04:38:18.4409317Z with: 2025-03-14T04:38:18.4409559Z env: 2025-03-14T04:38:18.4409811Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:18.4410186Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:18.4410588Z ##[endgroup] 2025-03-14T04:38:18.4445396Z ##[group]Run seemethere/download-artifact-s3@v4 2025-03-14T04:38:18.4445813Z with: 2025-03-14T04:38:18.4446180Z name: td_results 2025-03-14T04:38:18.4446472Z s3-bucket: gha-artifacts 2025-03-14T04:38:18.4446795Z region: us-east-1 2025-03-14T04:38:18.4447074Z env: 2025-03-14T04:38:18.4447332Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:18.4462195Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:18.4462714Z ##[endgroup] 2025-03-14T04:38:18.9241827Z (node:750433) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-03-14T04:38:18.9242458Z 2025-03-14T04:38:18.9242708Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-03-14T04:38:18.9243362Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-03-14T04:38:18.9244050Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-03-14T04:38:19.0345741Z Found 1 objects with prefix pytorch/pytorch/13849515290/td_results/ 2025-03-14T04:38:19.0346522Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2025-03-14T04:38:19.1018044Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2025-03-14T04:38:19.1024025Z Artifact download has finished successfully 2025-03-14T04:38:19.1219574Z ##[group]Run mkdir -p .additional_ci_files 2025-03-14T04:38:19.1220010Z mkdir -p .additional_ci_files 2025-03-14T04:38:19.1220541Z mv td_results.json .additional_ci_files/td_results.json || true 2025-03-14T04:38:19.1227324Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:38:19.1227773Z env: 2025-03-14T04:38:19.1228039Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:19.1228433Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:19.1228858Z ##[endgroup] 2025-03-14T04:38:19.1311689Z ##[group]Run .github/scripts/parse_ref.py 2025-03-14T04:38:19.1312129Z .github/scripts/parse_ref.py 2025-03-14T04:38:19.1318491Z shell: /usr/bin/bash -e {0} 2025-03-14T04:38:19.1318842Z env: 2025-03-14T04:38:19.1319106Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:19.1319527Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:19.1319980Z ##[endgroup] 2025-03-14T04:38:19.1675213Z Prepare all required actions 2025-03-14T04:38:19.1676071Z Getting action download info 2025-03-14T04:38:19.2774592Z ##[group]Run ./.github/actions/filter-test-configs 2025-03-14T04:38:19.2775019Z with: 2025-03-14T04:38:19.2775490Z github-token: *** 2025-03-14T04:38:19.2776891Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}]} 2025-03-14T04:38:19.2778637Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T04:38:19.2779294Z env: 2025-03-14T04:38:19.2779563Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:19.2779973Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:19.2780411Z ##[endgroup] 2025-03-14T04:38:19.2823873Z ##[group]Run nick-fields/retry@v3.0.0 2025-03-14T04:38:19.2824225Z with: 2025-03-14T04:38:19.2824473Z shell: bash 2025-03-14T04:38:19.2824735Z timeout_minutes: 10 2025-03-14T04:38:19.2825006Z max_attempts: 5 2025-03-14T04:38:19.2825284Z retry_wait_seconds: 30 2025-03-14T04:38:19.2826136Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2025-03-14T04:38:19.2827051Z polling_interval_seconds: 1 2025-03-14T04:38:19.2827373Z warning_on_retry: true 2025-03-14T04:38:19.2827670Z continue_on_error: false 2025-03-14T04:38:19.2827963Z env: 2025-03-14T04:38:19.2828207Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:19.2828576Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:19.2829147Z GITHUB_TOKEN: *** 2025-03-14T04:38:19.2829420Z ##[endgroup] 2025-03-14T04:38:19.3830773Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2025-03-14T04:38:19.6316755Z Defaulting to user installation because normal site-packages is not writeable 2025-03-14T04:38:19.6482921Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2025-03-14T04:38:19.6487742Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2025-03-14T04:38:19.6613049Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-03-14T04:38:19.6614783Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2025.1.31) 2025-03-14T04:38:19.6624130Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2025-03-14T04:38:19.6630127Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-03-14T04:38:20.3653873Z Command completed after 1 attempt(s). 2025-03-14T04:38:20.3719601Z ##[group]Run set -x 2025-03-14T04:38:20.3720012Z set -x 2025-03-14T04:38:20.3720326Z  2025-03-14T04:38:20.3720830Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-03-14T04:38:20.3721639Z # in runner workspace 2025-03-14T04:38:20.3722149Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-03-14T04:38:20.3730161Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:38:20.3730613Z env: 2025-03-14T04:38:20.3730893Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:20.3731293Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:20.3731730Z ##[endgroup] 2025-03-14T04:38:20.3761057Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-03-14T04:38:20.4030362Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-03-14T04:38:20.4031033Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-03-14T04:38:20.4031439Z echo "Job name: ${JOB_NAME}" 2025-03-14T04:38:20.4031788Z  2025-03-14T04:38:20.4032226Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-03-14T04:38:20.4032773Z # in runner workspace 2025-03-14T04:38:20.4033595Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-03-14T04:38:20.4034204Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-03-14T04:38:20.4034642Z  --job-name "${JOB_NAME}" \ 2025-03-14T04:38:20.4036193Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}]}" \ 2025-03-14T04:38:20.4037792Z  --selected-test-configs "" \ 2025-03-14T04:38:20.4038223Z  --pr-number "${PR_NUMBER}" \ 2025-03-14T04:38:20.4038615Z  --tag "${TAG}" \ 2025-03-14T04:38:20.4038999Z  --event-name "${EVENT_NAME}" \ 2025-03-14T04:38:20.4039651Z  --schedule "${SCHEDULE}" \ 2025-03-14T04:38:20.4040024Z  --branch "${HEAD_BRANCH}" 2025-03-14T04:38:20.4045885Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:38:20.4046324Z env: 2025-03-14T04:38:20.4046594Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:20.4046985Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:20.4047614Z GITHUB_TOKEN: *** 2025-03-14T04:38:20.4048173Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T04:38:20.4048796Z PR_NUMBER: 2025-03-14T04:38:20.4049069Z TAG: 2025-03-14T04:38:20.4049330Z EVENT_NAME: push 2025-03-14T04:38:20.4049618Z SCHEDULE: 2025-03-14T04:38:20.4049901Z HEAD_BRANCH: 2025-03-14T04:38:20.4050179Z ##[endgroup] 2025-03-14T04:38:20.4075830Z Workflow: pull 2025-03-14T04:38:20.4076501Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T04:38:20.6053395Z ##[group]Run echo "Filtered matrix:" 2025-03-14T04:38:20.6053831Z echo "Filtered matrix:" 2025-03-14T04:38:20.6055348Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu"}]}" 2025-03-14T04:38:20.6056662Z  2025-03-14T04:38:20.6056905Z echo 2025-03-14T04:38:20.6057365Z echo "Is the current job unstable? False" 2025-03-14T04:38:20.6057742Z  2025-03-14T04:38:20.6057963Z echo 2025-03-14T04:38:20.6058270Z echo "Is keep-going label set? False" 2025-03-14T04:38:20.6058628Z  2025-03-14T04:38:20.6058863Z echo 2025-03-14T04:38:20.6059132Z echo "Renabled issues? " 2025-03-14T04:38:20.6064647Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:38:20.6065206Z env: 2025-03-14T04:38:20.6065453Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:20.6065824Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:20.6066225Z ##[endgroup] 2025-03-14T04:38:20.6089868Z Filtered matrix: 2025-03-14T04:38:20.6091215Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu}]} 2025-03-14T04:38:20.6092593Z 2025-03-14T04:38:20.6092746Z Is the current job unstable? False 2025-03-14T04:38:20.6092988Z 2025-03-14T04:38:20.6093307Z Is keep-going label set? False 2025-03-14T04:38:20.6093536Z 2025-03-14T04:38:20.6093669Z Renabled issues? 2025-03-14T04:38:20.6157615Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-03-14T04:38:20.6158263Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-03-14T04:38:20.6164172Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T04:38:20.6164648Z env: 2025-03-14T04:38:20.6164923Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:20.6165445Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:20.6165980Z JOB_TIMEOUT: 360 2025-03-14T04:38:20.6166242Z ##[endgroup] 2025-03-14T04:38:20.6240434Z ##[group]Run set -x 2025-03-14T04:38:20.6240866Z set -x 2025-03-14T04:38:20.6241159Z  2025-03-14T04:38:20.6241483Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-03-14T04:38:20.6241977Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-03-14T04:38:20.6242474Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-03-14T04:38:20.6242930Z  TEST_COMMAND=.ci/onnx/test.sh 2025-03-14T04:38:20.6243314Z else 2025-03-14T04:38:20.6243633Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-03-14T04:38:20.6244013Z fi 2025-03-14T04:38:20.6244279Z  2025-03-14T04:38:20.6244629Z # Leaving 1GB for the runner and other things 2025-03-14T04:38:20.6245342Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-03-14T04:38:20.6246419Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-03-14T04:38:20.6247190Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-03-14T04:38:20.6247773Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-03-14T04:38:20.6248230Z  2025-03-14T04:38:20.6248541Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-03-14T04:38:20.6248925Z  SHM_OPTS= 2025-03-14T04:38:20.6249210Z  JENKINS_USER= 2025-03-14T04:38:20.6249603Z  # ensure that docker container cleanly exits in 12 hours 2025-03-14T04:38:20.6250132Z  # if for some reason cleanup action doesn't stop container 2025-03-14T04:38:20.6250584Z  # when job is cancelled 2025-03-14T04:38:20.6250936Z  DOCKER_SHELL_CMD="sleep 12h" 2025-03-14T04:38:20.6251272Z  2025-03-14T04:38:20.6251682Z  # since some steps are skipped on s390x, if they are necessary, run them here 2025-03-14T04:38:20.6252281Z  env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T04:38:20.6252774Z  env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T04:38:20.6253174Z else 2025-03-14T04:38:20.6253458Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-03-14T04:38:20.6253835Z  JENKINS_USER="--user jenkins" 2025-03-14T04:38:20.6254190Z  DOCKER_SHELL_CMD= 2025-03-14T04:38:20.6254483Z fi 2025-03-14T04:38:20.6254726Z  2025-03-14T04:38:20.6255102Z # detached container should get cleaned up by teardown_ec2_linux 2025-03-14T04:38:20.6255685Z # TODO: Stop building test binaries as part of the build phase 2025-03-14T04:38:20.6256492Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-03-14T04:38:20.6257081Z # shellcheck disable=SC2086,SC2090 2025-03-14T04:38:20.6257445Z container_name=$(docker run \ 2025-03-14T04:38:20.6257792Z  ${GPU_FLAG:-} \ 2025-03-14T04:38:20.6258135Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-03-14T04:38:20.6258521Z  -e BUILD_ENVIRONMENT \ 2025-03-14T04:38:20.6258855Z  -e PR_NUMBER \ 2025-03-14T04:38:20.6259158Z  -e GITHUB_ACTIONS \ 2025-03-14T04:38:20.6259479Z  -e GITHUB_REPOSITORY \ 2025-03-14T04:38:20.6259805Z  -e GITHUB_WORKFLOW \ 2025-03-14T04:38:20.6260117Z  -e GITHUB_JOB \ 2025-03-14T04:38:20.6260407Z  -e GITHUB_RUN_ID \ 2025-03-14T04:38:20.6260718Z  -e GITHUB_RUN_NUMBER \ 2025-03-14T04:38:20.6261044Z  -e GITHUB_RUN_ATTEMPT \ 2025-03-14T04:38:20.6261358Z  -e JOB_ID \ 2025-03-14T04:38:20.6261640Z  -e JOB_NAME \ 2025-03-14T04:38:20.6261920Z  -e BASE_SHA \ 2025-03-14T04:38:20.6262198Z  -e BRANCH \ 2025-03-14T04:38:20.6262473Z  -e SHA1 \ 2025-03-14T04:38:20.6262749Z  -e AWS_DEFAULT_REGION \ 2025-03-14T04:38:20.6263170Z  -e IN_WHEEL_TEST \ 2025-03-14T04:38:20.6263477Z  -e SHARD_NUMBER \ 2025-03-14T04:38:20.6263777Z  -e TEST_CONFIG \ 2025-03-14T04:38:20.6264080Z  -e NUM_TEST_SHARDS \ 2025-03-14T04:38:20.6264389Z  -e REENABLED_ISSUES \ 2025-03-14T04:38:20.6264714Z  -e CONTINUE_THROUGH_ERROR \ 2025-03-14T04:38:20.6265061Z  -e VERBOSE_TEST_LOGS \ 2025-03-14T04:38:20.6265380Z  -e TEST_SHOWLOCALS \ 2025-03-14T04:38:20.6265690Z  -e NO_TEST_TIMEOUT \ 2025-03-14T04:38:20.6265990Z  -e NO_TD \ 2025-03-14T04:38:20.6266265Z  -e TD_DISTRIBUTED \ 2025-03-14T04:38:20.6266565Z  -e PR_LABELS \ 2025-03-14T04:38:20.6266886Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-03-14T04:38:20.6267244Z  -e SCCACHE_BUCKET \ 2025-03-14T04:38:20.6267554Z  -e SCCACHE_REGION \ 2025-03-14T04:38:20.6267851Z  -e XLA_CUDA \ 2025-03-14T04:38:20.6268167Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-03-14T04:38:20.6268557Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-03-14T04:38:20.6268955Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-03-14T04:38:20.6269517Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-03-14T04:38:20.6270180Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-03-14T04:38:20.6270573Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-03-14T04:38:20.6271065Z  -e DASHBOARD_TAG \ 2025-03-14T04:38:20.6271404Z  -e IS_A100_RUNNER \ 2025-03-14T04:38:20.6271757Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-03-14T04:38:20.6272187Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-03-14T04:38:20.6272685Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-03-14T04:38:20.6273428Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-03-14T04:38:20.6273911Z  --security-opt seccomp=unconfined \ 2025-03-14T04:38:20.6274329Z  --cap-add=SYS_PTRACE \ 2025-03-14T04:38:20.6274695Z  --ipc=host \ 2025-03-14T04:38:20.6275014Z  ${SHM_OPTS} \ 2025-03-14T04:38:20.6275327Z  --tty \ 2025-03-14T04:38:20.6275623Z  --detach \ 2025-03-14T04:38:20.6275951Z  --name="${container_name}" \ 2025-03-14T04:38:20.6276337Z  ${JENKINS_USER} \ 2025-03-14T04:38:20.6276763Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-03-14T04:38:20.6277248Z  -w /var/lib/jenkins/workspace \ 2025-03-14T04:38:20.6277640Z  "${DOCKER_IMAGE}" \ 2025-03-14T04:38:20.6277976Z  ${DOCKER_SHELL_CMD} 2025-03-14T04:38:20.6278306Z ) 2025-03-14T04:38:20.6278658Z # Propagate download.pytorch.org IP to container 2025-03-14T04:38:20.6279594Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2025-03-14T04:38:20.6280454Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-03-14T04:38:20.6280953Z  2025-03-14T04:38:20.6281278Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-03-14T04:38:20.6281973Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-03-14T04:38:20.6282594Z fi 2025-03-14T04:38:20.6282849Z  2025-03-14T04:38:20.6283447Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-03-14T04:38:20.6289231Z shell: /usr/bin/bash -e {0} 2025-03-14T04:38:20.6289526Z env: 2025-03-14T04:38:20.6289752Z GIT_DEFAULT_BRANCH: main 2025-03-14T04:38:20.6290116Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:20.6290587Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2025-03-14T04:38:20.6290972Z PR_NUMBER: 2025-03-14T04:38:20.6291233Z GITHUB_REPOSITORY: pytorch/pytorch 2025-03-14T04:38:20.6291563Z GITHUB_WORKFLOW: pull 2025-03-14T04:38:20.6291830Z GITHUB_JOB: test 2025-03-14T04:38:20.6292219Z GITHUB_RUN_ID: 13849515290 2025-03-14T04:38:20.6292520Z GITHUB_RUN_NUMBER: 299021 2025-03-14T04:38:20.6292813Z GITHUB_RUN_ATTEMPT: 1 2025-03-14T04:38:20.6293080Z JOB_ID: 38755323799 2025-03-14T04:38:20.6293615Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T04:38:20.6294204Z BRANCH: main 2025-03-14T04:38:20.6294483Z SHA1: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:20.6294882Z BASE_SHA: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:20.6295254Z TEST_CONFIG: distributed 2025-03-14T04:38:20.6295537Z SHARD_NUMBER: 1 2025-03-14T04:38:20.6295790Z NUM_TEST_SHARDS: 3 2025-03-14T04:38:20.6296060Z REENABLED_ISSUES: 2025-03-14T04:38:20.6296351Z CONTINUE_THROUGH_ERROR: False 2025-03-14T04:38:20.6296674Z VERBOSE_TEST_LOGS: False 2025-03-14T04:38:20.6296979Z TEST_SHOWLOCALS: False 2025-03-14T04:38:20.6297276Z NO_TEST_TIMEOUT: False 2025-03-14T04:38:20.6297561Z NO_TD: False 2025-03-14T04:38:20.6297825Z TD_DISTRIBUTED: False 2025-03-14T04:38:20.6298176Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-03-14T04:38:20.6298577Z SCCACHE_REGION: us-east-1 2025-03-14T04:38:20.6298874Z SHM_SIZE: 2g 2025-03-14T04:38:20.6299633Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:38:20.6300460Z XLA_CUDA: 2025-03-14T04:38:20.6300856Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-03-14T04:38:20.6301357Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-03-14T04:38:20.6301717Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-03-14T04:38:20.6302062Z DASHBOARD_TAG: 2025-03-14T04:38:20.6302512Z HUGGING_FACE_HUB_TOKEN: *** 2025-03-14T04:38:20.6302968Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-03-14T04:38:20.6303299Z IS_A100_RUNNER: 0 2025-03-14T04:38:20.6303800Z ARTIFACTS_FILE_SUFFIX: test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799 2025-03-14T04:38:20.6304379Z ##[endgroup] 2025-03-14T04:38:20.6326818Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2025-03-14T04:38:20.6327304Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2025-03-14T04:38:20.6327741Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-03-14T04:38:20.6330625Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-03-14T04:38:20.6351284Z + TOTAL_AVAILABLE_MEMORY_IN_GB='185.682 ' 2025-03-14T04:38:20.6351896Z + TOTAL_MEMORY_WITH_SWAP=188 2025-03-14T04:38:20.6352318Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *\s\3\9\0\x* ]] 2025-03-14T04:38:20.6352765Z + SHM_OPTS=--shm-size=2g 2025-03-14T04:38:20.6353212Z + JENKINS_USER='--user jenkins' 2025-03-14T04:38:20.6353892Z + DOCKER_SHELL_CMD= 2025-03-14T04:38:20.6362103Z +++ nproc --ignore=2 2025-03-14T04:38:20.6375918Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --memory=185g --memory-swap=188g --env-file=/tmp/github_env_13849515290 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T04:38:32.5492703Z + container_name=1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T04:38:32.5497635Z + grep download.pytorch.org /etc/hosts 2025-03-14T04:38:32.5499318Z + docker exec -i 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 sudo bash -c '/bin/cat >> /etc/hosts' 2025-03-14T04:38:32.6899489Z + echo DOCKER_CONTAINER_ID=1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T04:38:32.6900277Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *\s\3\9\0\x* ]] 2025-03-14T04:38:32.6901728Z ++ echo dist/torch-2.8.0a0+gitaed0b7a-cp310-cp310-linux_x86_64.whl 2025-03-14T04:38:32.6904019Z + docker exec -t 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 sh -c 'python3 -m pip install dist/torch-2.8.0a0+gitaed0b7a-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-03-14T04:38:33.1534786Z Processing ./dist/torch-2.8.0a0+gitaed0b7a-cp310-cp310-linux_x86_64.whl (from torch==2.8.0a0+gitaed0b7a) 2025-03-14T04:38:34.1036809Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (3.16.1) 2025-03-14T04:38:34.1038489Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (4.12.2) 2025-03-14T04:38:34.1041306Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (1.13.3) 2025-03-14T04:38:34.1044693Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (2.8.8) 2025-03-14T04:38:34.1048133Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (3.1.6) 2025-03-14T04:38:34.1051564Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (2024.10.0) 2025-03-14T04:38:34.1071595Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (3.3.0) 2025-03-14T04:38:34.1093408Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (1.22.4) 2025-03-14T04:38:34.1110769Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (1.3.0) 2025-03-14T04:38:34.1495892Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (3.0.2) 2025-03-14T04:38:34.5187388Z Installing collected packages: torch 2025-03-14T04:38:45.8435677Z Successfully installed torch-2.8.0a0+gitaed0b7a 2025-03-14T04:38:45.9354432Z + export TERM=vt100 2025-03-14T04:38:45.9354828Z + TERM=vt100 2025-03-14T04:38:45.9359207Z ++ dirname .ci/pytorch/test.sh 2025-03-14T04:38:45.9359596Z + source .ci/pytorch/common.sh 2025-03-14T04:38:45.9362319Z +++ dirname .ci/pytorch/common.sh 2025-03-14T04:38:45.9370797Z ++ source .ci/pytorch/common_utils.sh 2025-03-14T04:38:45.9372377Z +++ declare -f -t trap_add 2025-03-14T04:38:45.9378638Z ++ set -ex -o pipefail 2025-03-14T04:38:45.9379017Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2025-03-14T04:38:45.9379480Z ++ BUILD_TEST_LIBTORCH=0 2025-03-14T04:38:45.9380219Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2025-03-14T04:38:45.9380686Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *s390x* ]] 2025-03-14T04:38:45.9381129Z + [[ -d /var/lib/jenkins/workspace ]] 2025-03-14T04:38:45.9384077Z ++ stat -c %u /var/lib/jenkins/workspace 2025-03-14T04:38:45.9399691Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-03-14T04:38:45.9400096Z + trap_add cleanup_workspace EXIT 2025-03-14T04:38:45.9400485Z + trap_add_cmd=cleanup_workspace 2025-03-14T04:38:45.9400837Z + shift 2025-03-14T04:38:45.9401103Z + for trap_add_name in "$@" 2025-03-14T04:38:45.9407023Z +++ trap -p EXIT 2025-03-14T04:38:45.9409679Z ++ eval 'extract_trap_cmd ' 2025-03-14T04:38:45.9410010Z +++ extract_trap_cmd 2025-03-14T04:38:45.9410313Z +++ printf '%s\n' '' 2025-03-14T04:38:45.9410630Z ++ printf '%s\n' cleanup_workspace 2025-03-14T04:38:45.9412065Z + trap -- ' 2025-03-14T04:38:45.9412349Z cleanup_workspace' EXIT 2025-03-14T04:38:45.9412735Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-03-14T04:38:46.4952326Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-03-14T04:38:46.4971752Z + echo 'Environment variables:' 2025-03-14T04:38:46.4972140Z Environment variables: 2025-03-14T04:38:46.4972445Z + env 2025-03-14T04:38:46.4979438Z INSTALLED_DB=yes 2025-03-14T04:38:46.4979789Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2025-03-14T04:38:46.4980150Z NVIDIA_VISIBLE_DEVICES=all 2025-03-14T04:38:46.4980672Z NV_NVML_DEV_VERSION=11.8.86-1 2025-03-14T04:38:46.4981183Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-03-14T04:38:46.4981840Z CONTINUE_THROUGH_ERROR=False 2025-03-14T04:38:46.4982240Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2025-03-14T04:38:46.4982691Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2025-03-14T04:38:46.4983135Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2025-03-14T04:38:46.4983554Z HOSTNAME=1e25b2eec068 2025-03-14T04:38:46.4984432Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.4985214Z GITHUB_ACTION=__self 2025-03-14T04:38:46.4985548Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-03-14T04:38:46.4987557Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2025-03-14T04:38:46.4989590Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2025-03-14T04:38:46.4990049Z NV_NVTX_VERSION=11.8.86-1 2025-03-14T04:38:46.4990386Z GITHUB_RUN_NUMBER=299021 2025-03-14T04:38:46.4990721Z TEST_CONFIG=distributed 2025-03-14T04:38:46.4991064Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-03-14T04:38:46.4991458Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-03-14T04:38:46.4992071Z IS_A100_RUNNER=0 2025-03-14T04:38:46.4992383Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2025-03-14T04:38:46.4992780Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2025-03-14T04:38:46.4993512Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-03-14T04:38:46.4993883Z NV_LIBNPP_VERSION=11.8.0.86-1 2025-03-14T04:38:46.4994263Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-03-14T04:38:46.4994727Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2025-03-14T04:38:46.4995171Z GITHUB_REF_TYPE=branch 2025-03-14T04:38:46.4995499Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-03-14T04:38:46.4995843Z NCCL_VERSION=2.16.2-1 2025-03-14T04:38:46.4996205Z BASE_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:46.4996615Z XLA_CUDA= 2025-03-14T04:38:46.4997028Z HUGGING_FACE_HUB_TOKEN=*** 2025-03-14T04:38:46.4997473Z *** 2025-03-14T04:38:46.4997749Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2025-03-14T04:38:46.4998127Z GITHUB_REPOSITORY_ID=65600975 2025-03-14T04:38:46.4998475Z GITHUB_ACTIONS=true 2025-03-14T04:38:46.4998792Z NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:46.4999204Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2025-03-14T04:38:46.4999646Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2025-03-14T04:38:46.5000079Z SHA1=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:46.5000508Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2025-03-14T04:38:46.5001085Z GITHUB_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:46.5001714Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/heads/main 2025-03-14T04:38:46.5002284Z UCC_HOME=/usr 2025-03-14T04:38:46.5002586Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2025-03-14T04:38:46.5002964Z VERBOSE_TEST_LOGS=False 2025-03-14T04:38:46.5003294Z NVIDIA_PRODUCT_NAME=CUDA 2025-03-14T04:38:46.5003677Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2025-03-14T04:38:46.5004110Z GITHUB_REF=refs/heads/main 2025-03-14T04:38:46.5004456Z NV_CUDA_CUDART_VERSION=11.8.89-1 2025-03-14T04:38:46.5004910Z SHARD_NUMBER=1 2025-03-14T04:38:46.5005198Z GITHUB_REF_PROTECTED=true 2025-03-14T04:38:46.5005520Z HOME=/var/lib/jenkins 2025-03-14T04:38:46.5005865Z GITHUB_API_URL=https://api.github.com 2025-03-14T04:38:46.5006256Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-03-14T04:38:46.5006678Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2025-03-14T04:38:46.5007096Z CUDA_VERSION=11.8.0 2025-03-14T04:38:46.5007439Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2025-03-14T04:38:46.5007838Z NUM_TEST_SHARDS=3 2025-03-14T04:38:46.5008119Z UCX_HOME=/usr 2025-03-14T04:38:46.5008532Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2025-03-14T04:38:46.5009434Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.5010480Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T04:38:46.5011498Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.5012447Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-03-14T04:38:46.5013057Z GITHUB_EVENT_NAME=push 2025-03-14T04:38:46.5013368Z DASHBOARD_TAG= 2025-03-14T04:38:46.5013664Z GITHUB_RUN_ID=13849515290 2025-03-14T04:38:46.5014040Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2025-03-14T04:38:46.5014499Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2025-03-14T04:38:46.5015311Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.5016123Z GITHUB_ACTOR=pytorchmergebot 2025-03-14T04:38:46.5016482Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2025-03-14T04:38:46.5016830Z PR_NUMBER= 2025-03-14T04:38:46.5017104Z GITHUB_RUN_ATTEMPT=1 2025-03-14T04:38:46.5017421Z ANACONDA_PYTHON_VERSION=3.10 2025-03-14T04:38:46.5017820Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-03-14T04:38:46.5018251Z TERM=vt100 2025-03-14T04:38:46.5018543Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2025-03-14T04:38:46.5018920Z INSTALLED_VISION=yes 2025-03-14T04:38:46.5019305Z BRANCH=main 2025-03-14T04:38:46.5019588Z SCCACHE_REGION=us-east-1 2025-03-14T04:38:46.5019920Z OPENSSL_ROOT_DIR=/opt/openssl 2025-03-14T04:38:46.5020285Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2025-03-14T04:38:46.5020674Z CUDA_PATH=/usr/local/cuda 2025-03-14T04:38:46.5021300Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-03-14T04:38:46.5021989Z GITHUB_SERVER_URL=https://github.com 2025-03-14T04:38:46.5022420Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2025-03-14T04:38:46.5022839Z REENABLED_ISSUES= 2025-03-14T04:38:46.5023120Z SHLVL=1 2025-03-14T04:38:46.5023393Z MAX_JOBS=46 2025-03-14T04:38:46.5023658Z NV_CUDA_LIB_VERSION=11.8.0-1 2025-03-14T04:38:46.5023984Z NVARCH=x86_64 2025-03-14T04:38:46.5024265Z GITHUB_ACTOR_ID=97764156 2025-03-14T04:38:46.5024673Z GITHUB_WORKFLOW_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:46.5025133Z GITHUB_REF_NAME=main 2025-03-14T04:38:46.5025465Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2025-03-14T04:38:46.5026004Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-03-14T04:38:46.5026515Z GITHUB_JOB=test 2025-03-14T04:38:46.5026833Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2025-03-14T04:38:46.5027327Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2025-03-14T04:38:46.5027864Z NO_TEST_TIMEOUT=False 2025-03-14T04:38:46.5028175Z TD_DISTRIBUTED=False 2025-03-14T04:38:46.5028507Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2025-03-14T04:38:46.5028912Z GITHUB_REPOSITORY=pytorch/pytorch 2025-03-14T04:38:46.5029285Z NV_NVPROF_VERSION=11.8.87-1 2025-03-14T04:38:46.5029626Z GITHUB_RETENTION_DAYS=90 2025-03-14T04:38:46.5029943Z OPENSSL_DIR=/opt/openssl 2025-03-14T04:38:46.5030272Z GITHUB_ACTION_REPOSITORY= 2025-03-14T04:38:46.5031190Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T04:38:46.5032137Z GITHUB_BASE_REF= 2025-03-14T04:38:46.5032680Z ARTIFACTS_FILE_SUFFIX=test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799 2025-03-14T04:38:46.5033589Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2025-03-14T04:38:46.5033943Z CI=true 2025-03-14T04:38:46.5034229Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2025-03-14T04:38:46.5034625Z GITHUB_REPOSITORY_OWNER=pytorch 2025-03-14T04:38:46.5034978Z JOB_ID=38755323799 2025-03-14T04:38:46.5035282Z INSTALLED_PROTOBUF=yes 2025-03-14T04:38:46.5035598Z GITHUB_HEAD_REF= 2025-03-14T04:38:46.5035890Z GITHUB_ACTION_REF= 2025-03-14T04:38:46.5036251Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-03-14T04:38:46.5036692Z TEST_SHOWLOCALS=False 2025-03-14T04:38:46.5037010Z GITHUB_WORKFLOW=pull 2025-03-14T04:38:46.5037339Z DEBIAN_FRONTEND=noninteractive 2025-03-14T04:38:46.5038082Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.5038855Z NO_TD=False 2025-03-14T04:38:46.5039148Z SKIP_SCCACHE_INITIALIZATION=1 2025-03-14T04:38:46.5039495Z _=/usr/bin/env 2025-03-14T04:38:46.5039895Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-03-14T04:38:46.5168397Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-03-14T04:38:46.5169181Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-03-14T04:38:46.5170255Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-03-14T04:38:46.5170984Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-03-14T04:38:46.5171547Z + BUILD_DIR=build 2025-03-14T04:38:46.5171866Z + BUILD_RENAMED_DIR=build_renamed 2025-03-14T04:38:46.5172243Z + BUILD_BIN_DIR=build/bin 2025-03-14T04:38:46.5172572Z + SHARD_NUMBER=1 2025-03-14T04:38:46.5172865Z + NUM_TEST_SHARDS=3 2025-03-14T04:38:46.5173191Z + export TORCH_SERIALIZATION_DEBUG=1 2025-03-14T04:38:46.5173584Z + TORCH_SERIALIZATION_DEBUG=1 2025-03-14T04:38:46.5173935Z + export VALGRIND=ON 2025-03-14T04:38:46.5174427Z + VALGRIND=ON 2025-03-14T04:38:46.5174779Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2025-03-14T04:38:46.5175266Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2025-03-14T04:38:46.5175741Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *s390x* ]] 2025-03-14T04:38:46.5176146Z + [[ 0 == \1 ]] 2025-03-14T04:38:46.5176432Z + [[ False == \1 ]] 2025-03-14T04:38:46.5176782Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2025-03-14T04:38:46.5177239Z ++ realpath build/custom_test_artifacts 2025-03-14T04:38:46.5183299Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-03-14T04:38:46.5183903Z + [[ -n '' ]] 2025-03-14T04:38:46.5184202Z + echo 'Environment variables' 2025-03-14T04:38:46.5184554Z Environment variables 2025-03-14T04:38:46.5184851Z + env 2025-03-14T04:38:46.5192073Z INSTALLED_DB=yes 2025-03-14T04:38:46.5192395Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2025-03-14T04:38:46.5192892Z NVIDIA_VISIBLE_DEVICES=all 2025-03-14T04:38:46.5193259Z NV_NVML_DEV_VERSION=11.8.86-1 2025-03-14T04:38:46.5193772Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-03-14T04:38:46.5194320Z CONTINUE_THROUGH_ERROR=False 2025-03-14T04:38:46.5194738Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2025-03-14T04:38:46.5195203Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2025-03-14T04:38:46.5195802Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2025-03-14T04:38:46.5196253Z HOSTNAME=1e25b2eec068 2025-03-14T04:38:46.5196959Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.5197730Z GITHUB_ACTION=__self 2025-03-14T04:38:46.5198067Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-03-14T04:38:46.5200002Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2025-03-14T04:38:46.5202029Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2025-03-14T04:38:46.5202482Z NV_NVTX_VERSION=11.8.86-1 2025-03-14T04:38:46.5202816Z GITHUB_RUN_NUMBER=299021 2025-03-14T04:38:46.5203154Z TEST_CONFIG=distributed 2025-03-14T04:38:46.5203483Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-03-14T04:38:46.5203888Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-03-14T04:38:46.5204274Z IS_A100_RUNNER=0 2025-03-14T04:38:46.5204582Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2025-03-14T04:38:46.5204975Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2025-03-14T04:38:46.5205567Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-03-14T04:38:46.5205934Z NV_LIBNPP_VERSION=11.8.0.86-1 2025-03-14T04:38:46.5206312Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-03-14T04:38:46.5206773Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2025-03-14T04:38:46.5207215Z GITHUB_REF_TYPE=branch 2025-03-14T04:38:46.5207548Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-03-14T04:38:46.5207888Z NCCL_VERSION=2.16.2-1 2025-03-14T04:38:46.5208361Z BASE_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:46.5208759Z XLA_CUDA= 2025-03-14T04:38:46.5209154Z HUGGING_FACE_HUB_TOKEN=*** 2025-03-14T04:38:46.5209697Z *** 2025-03-14T04:38:46.5209979Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2025-03-14T04:38:46.5210353Z GITHUB_REPOSITORY_ID=65600975 2025-03-14T04:38:46.5210698Z GITHUB_ACTIONS=true 2025-03-14T04:38:46.5211011Z NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T04:38:46.5211397Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2025-03-14T04:38:46.5211886Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2025-03-14T04:38:46.5212309Z SHA1=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:46.5212726Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2025-03-14T04:38:46.5213154Z GITHUB_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:46.5213744Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/heads/main 2025-03-14T04:38:46.5214390Z UCC_HOME=/usr 2025-03-14T04:38:46.5214685Z TORCH_SERIALIZATION_DEBUG=1 2025-03-14T04:38:46.5215036Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2025-03-14T04:38:46.5215405Z VERBOSE_TEST_LOGS=False 2025-03-14T04:38:46.5215723Z NVIDIA_PRODUCT_NAME=CUDA 2025-03-14T04:38:46.5216105Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2025-03-14T04:38:46.5216527Z GITHUB_REF=refs/heads/main 2025-03-14T04:38:46.5216862Z NV_CUDA_CUDART_VERSION=11.8.89-1 2025-03-14T04:38:46.5217205Z SHARD_NUMBER=1 2025-03-14T04:38:46.5217493Z GITHUB_REF_PROTECTED=true 2025-03-14T04:38:46.5217817Z HOME=/var/lib/jenkins 2025-03-14T04:38:46.5218152Z GITHUB_API_URL=https://api.github.com 2025-03-14T04:38:46.5218553Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-03-14T04:38:46.5218973Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2025-03-14T04:38:46.5219389Z CUDA_VERSION=11.8.0 2025-03-14T04:38:46.5219715Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2025-03-14T04:38:46.5220113Z NUM_TEST_SHARDS=3 2025-03-14T04:38:46.5220398Z UCX_HOME=/usr 2025-03-14T04:38:46.5220806Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2025-03-14T04:38:46.5221697Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.5222816Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T04:38:46.5223834Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.5224784Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-03-14T04:38:46.5225387Z GITHUB_EVENT_NAME=push 2025-03-14T04:38:46.5225697Z DASHBOARD_TAG= 2025-03-14T04:38:46.5225982Z GITHUB_RUN_ID=13849515290 2025-03-14T04:38:46.5226354Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2025-03-14T04:38:46.5226799Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2025-03-14T04:38:46.5227609Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.5228428Z GITHUB_ACTOR=pytorchmergebot 2025-03-14T04:38:46.5228781Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2025-03-14T04:38:46.5229126Z PR_NUMBER= 2025-03-14T04:38:46.5229393Z GITHUB_RUN_ATTEMPT=1 2025-03-14T04:38:46.5229689Z VALGRIND=ON 2025-03-14T04:38:46.5229981Z ANACONDA_PYTHON_VERSION=3.10 2025-03-14T04:38:46.5230377Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-03-14T04:38:46.5230802Z TERM=vt100 2025-03-14T04:38:46.5231096Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2025-03-14T04:38:46.5231474Z INSTALLED_VISION=yes 2025-03-14T04:38:46.5231772Z BRANCH=main 2025-03-14T04:38:46.5232052Z SCCACHE_REGION=us-east-1 2025-03-14T04:38:46.5232389Z OPENSSL_ROOT_DIR=/opt/openssl 2025-03-14T04:38:46.5232761Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2025-03-14T04:38:46.5233424Z CUDA_PATH=/usr/local/cuda 2025-03-14T04:38:46.5234066Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-03-14T04:38:46.5234786Z GITHUB_SERVER_URL=https://github.com 2025-03-14T04:38:46.5235236Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2025-03-14T04:38:46.5235672Z REENABLED_ISSUES= 2025-03-14T04:38:46.5235964Z SHLVL=1 2025-03-14T04:38:46.5236226Z MAX_JOBS=46 2025-03-14T04:38:46.5236503Z NV_CUDA_LIB_VERSION=11.8.0-1 2025-03-14T04:38:46.5236840Z NVARCH=x86_64 2025-03-14T04:38:46.5237133Z GITHUB_ACTOR_ID=97764156 2025-03-14T04:38:46.5237560Z GITHUB_WORKFLOW_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T04:38:46.5238030Z GITHUB_REF_NAME=main 2025-03-14T04:38:46.5238371Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2025-03-14T04:38:46.5238917Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-03-14T04:38:46.5239443Z GITHUB_JOB=test 2025-03-14T04:38:46.5239770Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2025-03-14T04:38:46.5240275Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2025-03-14T04:38:46.5240829Z NO_TEST_TIMEOUT=False 2025-03-14T04:38:46.5241144Z TD_DISTRIBUTED=False 2025-03-14T04:38:46.5241488Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2025-03-14T04:38:46.5241897Z GITHUB_REPOSITORY=pytorch/pytorch 2025-03-14T04:38:46.5242276Z NV_NVPROF_VERSION=11.8.87-1 2025-03-14T04:38:46.5242620Z GITHUB_RETENTION_DAYS=90 2025-03-14T04:38:46.5242947Z OPENSSL_DIR=/opt/openssl 2025-03-14T04:38:46.5243286Z GITHUB_ACTION_REPOSITORY= 2025-03-14T04:38:46.5244235Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T04:38:46.5245314Z GITHUB_BASE_REF= 2025-03-14T04:38:46.5245857Z ARTIFACTS_FILE_SUFFIX=test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799 2025-03-14T04:38:46.5246496Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2025-03-14T04:38:46.5246835Z CI=true 2025-03-14T04:38:46.5247108Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2025-03-14T04:38:46.5247484Z GITHUB_REPOSITORY_OWNER=pytorch 2025-03-14T04:38:46.5247828Z JOB_ID=38755323799 2025-03-14T04:38:46.5248118Z INSTALLED_PROTOBUF=yes 2025-03-14T04:38:46.5248426Z GITHUB_HEAD_REF= 2025-03-14T04:38:46.5248711Z GITHUB_ACTION_REF= 2025-03-14T04:38:46.5249066Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-03-14T04:38:46.5249490Z TEST_SHOWLOCALS=False 2025-03-14T04:38:46.5249870Z GITHUB_WORKFLOW=pull 2025-03-14T04:38:46.5250199Z DEBIAN_FRONTEND=noninteractive 2025-03-14T04:38:46.5250925Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_00a19553-b318-4957-be54-5fff128ce912 2025-03-14T04:38:46.5251672Z NO_TD=False 2025-03-14T04:38:46.5251965Z SKIP_SCCACHE_INITIALIZATION=1 2025-03-14T04:38:46.5252304Z _=/usr/bin/env 2025-03-14T04:38:46.5252591Z + echo 'Testing pytorch' 2025-03-14T04:38:46.5252909Z Testing pytorch 2025-03-14T04:38:46.5253203Z + export LANG=C.UTF-8 2025-03-14T04:38:46.5253499Z + LANG=C.UTF-8 2025-03-14T04:38:46.5253771Z + PR_NUMBER= 2025-03-14T04:38:46.5254061Z + [[ distributed == \d\e\f\a\u\l\t ]] 2025-03-14T04:38:46.5254457Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-03-14T04:38:46.5254897Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2025-03-14T04:38:46.5255319Z + [[ distributed == \s\l\o\w ]] 2025-03-14T04:38:46.5255748Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2025-03-14T04:38:46.5256253Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2025-03-14T04:38:46.5256706Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-03-14T04:38:46.5257113Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-03-14T04:38:46.5257497Z + [[ distributed == *crossref* ]] 2025-03-14T04:38:46.5257893Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2025-03-14T04:38:46.5258347Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2025-03-14T04:38:46.5258813Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2025-03-14T04:38:46.5259252Z + pip_install --user ninja==1.10.2 2025-03-14T04:38:46.5259717Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T04:38:46.5260302Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2025-03-14T04:38:48.0505446Z Collecting ninja==1.10.2 2025-03-14T04:38:48.0924057Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-03-14T04:38:48.1075663Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-03-14T04:38:48.4901503Z Installing collected packages: ninja 2025-03-14T04:38:48.4977405Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2025-03-14T04:38:48.4978564Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2025-03-14T04:38:48.5051126Z Successfully installed ninja-1.10.2 2025-03-14T04:38:48.5918288Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T04:38:48.5920513Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T04:38:48.5921704Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2025-03-14T04:38:48.5922149Z + install_tlparse 2025-03-14T04:38:48.5922492Z + pip_install --user tlparse==0.3.30 2025-03-14T04:38:48.5922972Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T04:38:48.5923581Z + python3 -m pip install --progress-bar off --user tlparse==0.3.30 2025-03-14T04:38:49.0821760Z Collecting tlparse==0.3.30 2025-03-14T04:38:49.1203506Z Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB) 2025-03-14T04:38:49.1334771Z Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB) 2025-03-14T04:38:49.5443159Z Installing collected packages: tlparse 2025-03-14T04:38:49.5834250Z Successfully installed tlparse-0.3.30 2025-03-14T04:38:49.6740296Z ++ python -m site --user-base 2025-03-14T04:38:49.6923917Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T04:38:49.6926492Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2025-03-14T04:38:49.6927019Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2025-03-14T04:38:49.6927506Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2025-03-14T04:38:49.6928177Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2025-03-14T04:38:49.6928991Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2025-03-14T04:38:49.6929564Z + cd test 2025-03-14T04:38:49.6929979Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-03-14T04:38:51.4539884Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-03-14T04:38:51.4540484Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-03-14T04:38:51.4541865Z + DYNAMO_BENCHMARK_FLAGS=() 2025-03-14T04:38:51.4542729Z + [[ distributed == *pr_time_benchmarks* ]] 2025-03-14T04:38:51.4543142Z + [[ distributed == *dynamo_eager* ]] 2025-03-14T04:38:51.4557313Z + [[ distributed == *aot_eager* ]] 2025-03-14T04:38:51.4557886Z + [[ distributed == *aot_inductor* ]] 2025-03-14T04:38:51.4558318Z + [[ distributed == *max_autotune_inductor* ]] 2025-03-14T04:38:51.4558748Z + [[ distributed == *inductor* ]] 2025-03-14T04:38:51.4559131Z + [[ distributed == *dynamic* ]] 2025-03-14T04:38:51.4559502Z + [[ distributed == *cpu* ]] 2025-03-14T04:38:51.4559876Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2025-03-14T04:38:51.4584409Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2025-03-14T04:38:51.4584952Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2025-03-14T04:38:51.4585666Z + cd test 2025-03-14T04:38:51.4586664Z + python -c 'import torch; print(torch.__config__.show())' 2025-03-14T04:38:53.4802549Z PyTorch built with: 2025-03-14T04:38:53.4802922Z - GCC 9.4 2025-03-14T04:38:53.4803214Z - C++ Version: 201703 2025-03-14T04:38:53.4803915Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2025-03-14T04:38:53.4804835Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-03-14T04:38:53.4805506Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-03-14T04:38:53.4805927Z - LAPACK is enabled (usually provided by MKL) 2025-03-14T04:38:53.4806338Z - NNPACK is enabled 2025-03-14T04:38:53.4806668Z - CPU capability usage: AVX512 2025-03-14T04:38:53.4807020Z - CUDA Runtime 11.8 2025-03-14T04:38:53.4807442Z - NVCC architecture flags: -gencode;arch=compute_75,code=sm_75 2025-03-14T04:38:53.4807921Z - CuDNN 90.1 2025-03-14T04:38:53.4808189Z - Magma 2.6.1 2025-03-14T04:38:53.4813607Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2025-03-14T04:38:53.4819363Z 2025-03-14T04:38:53.9164824Z + cd test 2025-03-14T04:38:53.9165460Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-03-14T04:38:55.4178917Z ATen/Parallel: 2025-03-14T04:38:55.4179337Z at::get_num_threads() : 24 2025-03-14T04:38:55.4180082Z at::get_num_interop_threads() : 24 2025-03-14T04:38:55.4180549Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-03-14T04:38:55.4180915Z omp_get_max_threads() : 24 2025-03-14T04:38:55.4181596Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2025-03-14T04:38:55.4182308Z mkl_get_max_threads() : 24 2025-03-14T04:38:55.4182777Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-03-14T04:38:55.4183322Z std::thread::hardware_concurrency() : 48 2025-03-14T04:38:55.4183720Z Environment variables: 2025-03-14T04:38:55.4184042Z OMP_NUM_THREADS : [not set] 2025-03-14T04:38:55.4184412Z MKL_NUM_THREADS : [not set] 2025-03-14T04:38:55.4184764Z ATen parallel backend: OpenMP 2025-03-14T04:38:55.4184988Z 2025-03-14T04:38:55.6705337Z + [[ distributed == *numpy_2* ]] 2025-03-14T04:38:55.6706335Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2025-03-14T04:38:55.6707003Z + [[ distributed == *backward* ]] 2025-03-14T04:38:55.6707432Z + [[ distributed == *xla* ]] 2025-03-14T04:38:55.6707798Z + [[ distributed == *executorch* ]] 2025-03-14T04:38:55.6708200Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2025-03-14T04:38:55.6708652Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2025-03-14T04:38:55.6709100Z + [[ distributed == distributed ]] 2025-03-14T04:38:55.6709461Z + test_distributed 2025-03-14T04:38:55.6709792Z + echo 'Testing distributed python tests' 2025-03-14T04:38:55.6710203Z Testing distributed python tests 2025-03-14T04:38:55.6710705Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2025-03-14T04:38:55.7796644Z /var/lib/jenkins/workspace/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2025-03-14T04:38:55.7797735Z import pkg_resources 2025-03-14T04:39:01.0309110Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-03-14T04:39:01.1075010Z Ignoring disabled issues: [''] 2025-03-14T04:39:01.1266875Z Found test times from artifacts 2025-03-14T04:39:01.2019434Z Found test times from artifacts 2025-03-14T04:39:01.2040960Z Running all tests 2025-03-14T04:39:01.2165096Z Running parallel tests on 2 processes 2025-03-14T04:39:01.2169062Z Name: tests to run (est. time: 124.08min) 2025-03-14T04:39:01.2169662Z Serial tests (77): 2025-03-14T04:39:01.2170017Z distributed/test_distributed_spawn 1/10 2025-03-14T04:39:01.2170453Z distributed/test_distributed_spawn 4/10 2025-03-14T04:39:01.2170886Z distributed/test_distributed_spawn 7/10 2025-03-14T04:39:01.2171600Z distributed/test_distributed_spawn 10/10 2025-03-14T04:39:01.2172008Z distributed/fsdp/test_fsdp_core 3/3 2025-03-14T04:39:01.2172460Z distributed/fsdp/test_distributed_checkpoint 1/1 2025-03-14T04:39:01.2172922Z distributed/fsdp/test_fsdp_misc 1/1 2025-03-14T04:39:01.2173319Z distributed/test_c10d_nccl 1/3 2025-03-14T04:39:01.2173735Z distributed/test_c10d_functional_native 1/1 2025-03-14T04:39:01.2174157Z distributed/test_c10d_pypg 1/1 2025-03-14T04:39:01.2174533Z distributed/test_c10d_logger 1/1 2025-03-14T04:39:01.2174946Z distributed/test_c10d_object_collectives 1/1 2025-03-14T04:39:01.2175366Z distributed/test_c10d_gloo 1/2 2025-03-14T04:39:01.2175753Z distributed/tensor/test_attention 1/1 2025-03-14T04:39:01.2176185Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2025-03-14T04:39:01.2176617Z distributed/fsdp/test_fsdp_fx 1/1 2025-03-14T04:39:01.2177041Z distributed/tensor/test_experimental_ops 1/1 2025-03-14T04:39:01.2177571Z distributed/tensor/experimental/test_register_sharding 1/1 2025-03-14T04:39:01.2178076Z distributed/test_backends 1/1 2025-03-14T04:39:01.2178486Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 2025-03-14T04:39:01.2178920Z distributed/tensor/test_init 1/1 2025-03-14T04:39:01.2179358Z distributed/checkpoint/test_fsdp_optim_state 1/1 2025-03-14T04:39:01.2179951Z distributed/checkpoint/test_save_load_api 1/1 2025-03-14T04:39:01.2180411Z distributed/_tools/test_memory_tracker 1/1 2025-03-14T04:39:01.2180840Z distributed/test_collective_utils 1/1 2025-03-14T04:39:01.2181258Z distributed/tensor/test_random_ops 1/1 2025-03-14T04:39:01.2181682Z distributed/pipelining/test_backward 1/1 2025-03-14T04:39:01.2182109Z distributed/pipelining/test_pipe 1/1 2025-03-14T04:39:01.2182512Z distributed/test_device_mesh 1/1 2025-03-14T04:39:01.2182968Z distributed/tensor/experimental/test_tp_transform 1/1 2025-03-14T04:39:01.2183442Z distributed/tensor/test_utils 1/1 2025-03-14T04:39:01.2183924Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2025-03-14T04:39:01.2184420Z distributed/pipelining/test_stage 1/1 2025-03-14T04:39:01.2184868Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2025-03-14T04:39:01.2185341Z distributed/test_compute_comm_reordering 1/1 2025-03-14T04:39:01.2185767Z distributed/test_nccl 1/1 2025-03-14T04:39:01.2186175Z distributed/checkpoint/test_fsdp_model_state 1/1 2025-03-14T04:39:01.2186675Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2025-03-14T04:39:01.2187143Z distributed/checkpoint/test_planner 1/1 2025-03-14T04:39:01.2187600Z distributed/checkpoint/test_dtensor_resharding 1/1 2025-03-14T04:39:01.2188096Z distributed/optim/test_named_optimizer 1/1 2025-03-14T04:39:01.2188525Z distributed/tensor/test_matrix_ops 1/1 2025-03-14T04:39:01.2188977Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 2025-03-14T04:39:01.2189433Z distributed/checkpoint/test_utils 1/1 2025-03-14T04:39:01.2189869Z distributed/tensor/parallel/test_tp_examples 1/1 2025-03-14T04:39:01.2190400Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2025-03-14T04:39:01.2190945Z distributed/_composable/fsdp/test_fully_shard_init 1/1 2025-03-14T04:39:01.2191435Z distributed/fsdp/test_fsdp_optim_state 1/1 2025-03-14T04:39:01.2191868Z distributed/fsdp/test_fsdp_uneven 1/1 2025-03-14T04:39:01.2192409Z distributed/_composable/test_composability/test_2d_composability 1/1 2025-03-14T04:39:01.2193084Z distributed/fsdp/test_fsdp_exec_order 1/1 2025-03-14T04:39:01.2193689Z distributed/test_dynamo_distributed 1/1 2025-03-14T04:39:01.2194188Z distributed/checkpoint/test_file_system_checkpoint 1/1 2025-03-14T04:39:01.2194724Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 2025-03-14T04:39:01.2195284Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 2025-03-14T04:39:01.2195798Z distributed/launcher/test_run 1/1 2025-03-14T04:39:01.2196239Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2025-03-14T04:39:01.2196869Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2025-03-14T04:39:01.2197440Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2025-03-14T04:39:01.2197928Z distributed/test_launcher 1/1 2025-03-14T04:39:01.2198350Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2025-03-14T04:39:01.2198853Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2025-03-14T04:39:01.2199328Z distributed/_shard/test_sharder 1/1 2025-03-14T04:39:01.2199735Z distributed/fsdp/test_wrap 1/1 2025-03-14T04:39:01.2200190Z distributed/optim/test_apply_optimizer_in_backward 1/1 2025-03-14T04:39:01.2200755Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2025-03-14T04:39:01.2201275Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2025-03-14T04:39:01.2201728Z distributed/test_c10d_ops_nccl 1/1 2025-03-14T04:39:01.2202161Z distributed/_tools/test_fake_collectives 1/1 2025-03-14T04:39:01.2202620Z distributed/elastic/events/lib_test 1/1 2025-03-14T04:39:01.2203061Z distributed/elastic/metrics/api_test 1/1 2025-03-14T04:39:01.2203535Z distributed/elastic/multiprocessing/api_test 1/1 2025-03-14T04:39:01.2204047Z distributed/elastic/timer/local_timer_example 1/1 2025-03-14T04:39:01.2204557Z distributed/elastic/timer/local_timer_test 1/1 2025-03-14T04:39:01.2205117Z distributed/elastic/utils/distributed_test 1/1 2025-03-14T04:39:01.2205699Z distributed/elastic/utils/logging_test 1/1 2025-03-14T04:39:01.2206134Z distributed/elastic/utils/util_test 1/1 2025-03-14T04:39:01.2206530Z Parallel tests (0): 2025-03-14T04:39:01.2206858Z Name: excluded (est. time: 0.0min) 2025-03-14T04:39:01.2207202Z Serial tests (0): 2025-03-14T04:39:01.2207501Z Parallel tests (0): 2025-03-14T04:39:01.2235081Z Running distributed/test_distributed_spawn 1/10 ... [2025-03-14 04:39:01.222930] 2025-03-14T04:39:01.2241009Z /usr/bin/mpiexec 2025-03-14T04:39:01.2242100Z MPI not available -- MPI backend tests will be skipped 2025-03-14T04:39:01.2243197Z Running distributed tests for the test backend with env init_method 2025-03-14T04:39:01.2245537Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T04:39:01.2249740Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 04:39:01.224793] 2025-03-14T04:39:06.0375061Z 2025-03-14T04:39:06.0376298Z distributed/test_distributed_spawn 1/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.10_871ab4422f46658d_.log 2025-03-14T04:39:06.0377561Z Running 0 items in this shard: 2025-03-14T04:39:06.0377910Z 2025-03-14T04:39:06.0386008Z Running distributed tests for the test backend with file init_method 2025-03-14T04:39:06.0387953Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T04:39:06.0392116Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 04:39:06.039018] 2025-03-14T04:39:10.5173274Z 2025-03-14T04:39:10.5174531Z distributed/test_distributed_spawn 1/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.10_e5fc3e94f7bc89d6_.log 2025-03-14T04:39:10.5175688Z Running 0 items in this shard: 2025-03-14T04:39:10.5175940Z 2025-03-14T04:39:10.5181964Z Running distributed tests for the nccl backend with env init_method 2025-03-14T04:39:10.5183799Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T04:39:10.5188273Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 04:39:10.518636] 2025-03-14T04:45:03.8406822Z 2025-03-14T04:45:03.8408146Z distributed/test_distributed_spawn 1/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.10_fcce9f74fd866ffb_.log 2025-03-14T04:45:03.8430344Z Running 40 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2025-03-14T04:45:03.8452985Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2025-03-14T04:45:03.8454258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2025-03-14T04:45:03.8455657Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2025-03-14T04:45:03.8457070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2025-03-14T04:45:03.8458469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2025-03-14T04:45:03.8459752Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2025-03-14T04:45:03.8461064Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2025-03-14T04:45:03.8462441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2025-03-14T04:45:03.8463811Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-03-14T04:45:03.8465065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-03-14T04:45:03.8466218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2025-03-14T04:45:03.8467404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2025-03-14T04:45:03.8468690Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-03-14T04:45:03.8470389Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2025-03-14T04:45:03.8471819Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2025-03-14T04:45:03.8473235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2025-03-14T04:45:03.8474460Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-03-14T04:45:03.8475743Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2025-03-14T04:45:03.8477016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2025-03-14T04:45:03.8478261Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2025-03-14T04:45:03.8479528Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-03-14T04:45:03.8480803Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2025-03-14T04:45:03.8482230Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2025-03-14T04:45:03.8483749Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-03-14T04:45:03.8485407Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2025-03-14T04:45:03.8488382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2025-03-14T04:45:03.8489593Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-03-14T04:45:03.8490707Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2025-03-14T04:45:03.8491814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2025-03-14T04:45:03.8493170Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2025-03-14T04:45:03.8494522Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2025-03-14T04:45:03.8495903Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2025-03-14T04:45:03.8497167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2025-03-14T04:45:03.8498259Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2025-03-14T04:45:03.8499437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2025-03-14T04:45:03.8500717Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-03-14T04:45:03.8501913Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-03-14T04:45:03.8503059Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2025-03-14T04:45:03.8504163Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-03-14T04:45:03.8505371Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2025-03-14T04:45:03.8506118Z 2025-03-14T04:45:03.8506374Z Running distributed tests for the nccl backend with file init_method 2025-03-14T04:45:03.8506896Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T04:45:03.8508218Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 04:45:03.842247] 2025-03-14T04:50:57.1542862Z 2025-03-14T04:50:57.1545580Z distributed/test_distributed_spawn 1/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.10_a98da89a2b5bf43e_.log 2025-03-14T04:50:57.1568495Z Running 40 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2025-03-14T04:50:57.1590923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2025-03-14T04:50:57.1592262Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2025-03-14T04:50:57.1593889Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2025-03-14T04:50:57.1595346Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2025-03-14T04:50:57.1596720Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2025-03-14T04:50:57.1598036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2025-03-14T04:50:57.1599399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2025-03-14T04:50:57.1600825Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2025-03-14T04:50:57.1602334Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-03-14T04:50:57.1603625Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-03-14T04:50:57.1604804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2025-03-14T04:50:57.1606210Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2025-03-14T04:50:57.1607475Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-03-14T04:50:57.1608773Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2025-03-14T04:50:57.1610118Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2025-03-14T04:50:57.1611367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2025-03-14T04:50:57.1612513Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-03-14T04:50:57.1615676Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2025-03-14T04:50:57.1616881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2025-03-14T04:50:57.1618065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2025-03-14T04:50:57.1619262Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-03-14T04:50:57.1620469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2025-03-14T04:50:57.1621703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2025-03-14T04:50:57.1623126Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-03-14T04:50:57.1624527Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2025-03-14T04:50:57.1627727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2025-03-14T04:50:57.1629047Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-03-14T04:50:57.1630170Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2025-03-14T04:50:57.1631471Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2025-03-14T04:50:57.1632725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2025-03-14T04:50:57.1634391Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2025-03-14T04:50:57.1635853Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2025-03-14T04:50:57.1637198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2025-03-14T04:50:57.1638441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2025-03-14T04:50:57.1639690Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2025-03-14T04:50:57.1641037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-03-14T04:50:57.1642306Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-03-14T04:50:57.1643529Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2025-03-14T04:50:57.1644704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-03-14T04:50:57.1646057Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2025-03-14T04:50:57.1646797Z 2025-03-14T04:50:57.1647059Z Running distributed tests for the gloo backend with env init_method 2025-03-14T04:50:57.1647557Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T04:50:57.1648861Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 04:50:57.155907] 2025-03-14T04:57:35.9758468Z 2025-03-14T04:57:35.9759987Z distributed/test_distributed_spawn 1/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.10_06c84ed7b4e19b0f_.log 2025-03-14T04:57:35.9784588Z Running 40 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2025-03-14T04:57:35.9807088Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2025-03-14T04:57:35.9808359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2025-03-14T04:57:35.9809762Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2025-03-14T04:57:35.9811167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2025-03-14T04:57:35.9812566Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2025-03-14T04:57:35.9813849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2025-03-14T04:57:35.9815158Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2025-03-14T04:57:35.9816542Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2025-03-14T04:57:35.9817915Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-03-14T04:57:35.9819166Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-03-14T04:57:35.9820313Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2025-03-14T04:57:35.9821497Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2025-03-14T04:57:35.9822842Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-03-14T04:57:35.9824177Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2025-03-14T04:57:35.9825557Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2025-03-14T04:57:35.9826847Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2025-03-14T04:57:35.9828030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-03-14T04:57:35.9829461Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2025-03-14T04:57:35.9830612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2025-03-14T04:57:35.9831713Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2025-03-14T04:57:35.9832840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-03-14T04:57:35.9834307Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2025-03-14T04:57:35.9835629Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2025-03-14T04:57:35.9837151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-03-14T04:57:35.9838650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2025-03-14T04:57:35.9840015Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2025-03-14T04:57:35.9841147Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-03-14T04:57:35.9842200Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2025-03-14T04:57:35.9843230Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2025-03-14T04:57:35.9844434Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2025-03-14T04:57:35.9845703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2025-03-14T04:57:35.9846996Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2025-03-14T04:57:35.9848188Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2025-03-14T04:57:35.9849400Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2025-03-14T04:57:35.9850571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2025-03-14T04:57:35.9851840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-03-14T04:57:35.9853080Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-03-14T04:57:35.9854230Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2025-03-14T04:57:35.9855423Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-03-14T04:57:35.9856562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2025-03-14T04:57:35.9857259Z 2025-03-14T04:57:35.9857498Z Running distributed tests for the gloo backend with file init_method 2025-03-14T04:57:35.9857986Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T04:57:35.9859227Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 04:57:35.978007] 2025-03-14T05:04:16.2793415Z 2025-03-14T05:04:16.2795098Z distributed/test_distributed_spawn 1/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.10_f0cafc9ee0c3eb5e_.log 2025-03-14T05:04:16.2817702Z Running 40 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2025-03-14T05:04:16.2840193Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2025-03-14T05:04:16.2841506Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2025-03-14T05:04:16.2842957Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2025-03-14T05:04:16.2844412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2025-03-14T05:04:16.2845857Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2025-03-14T05:04:16.2847132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2025-03-14T05:04:16.2848447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2025-03-14T05:04:16.2849833Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2025-03-14T05:04:16.2851317Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-03-14T05:04:16.2852578Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-03-14T05:04:16.2853724Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2025-03-14T05:04:16.2854910Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2025-03-14T05:04:16.2856195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-03-14T05:04:16.2857527Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2025-03-14T05:04:16.2858911Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2025-03-14T05:04:16.2860256Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2025-03-14T05:04:16.2861436Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-03-14T05:04:16.2862666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2025-03-14T05:04:16.2863893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2025-03-14T05:04:16.2865094Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2025-03-14T05:04:16.2866328Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-03-14T05:04:16.2867567Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2025-03-14T05:04:16.2868838Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2025-03-14T05:04:16.2870679Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-03-14T05:04:16.2872168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2025-03-14T05:04:16.2873534Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2025-03-14T05:04:16.2874814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-03-14T05:04:16.2875995Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2025-03-14T05:04:16.2877162Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2025-03-14T05:04:16.2889789Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2025-03-14T05:04:16.2891156Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2025-03-14T05:04:16.2892539Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2025-03-14T05:04:16.2894004Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2025-03-14T05:04:16.2895101Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2025-03-14T05:04:16.2896283Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2025-03-14T05:04:16.2897557Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-03-14T05:04:16.2898761Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-03-14T05:04:16.2899914Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2025-03-14T05:04:16.2901008Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-03-14T05:04:16.2902222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2025-03-14T05:04:16.2902959Z 2025-03-14T05:04:16.2903354Z Running distributed/test_distributed_spawn 4/10 ... [2025-03-14 05:04:16.280961] 2025-03-14T05:04:16.2903923Z /usr/bin/mpiexec 2025-03-14T05:04:16.2904291Z MPI not available -- MPI backend tests will be skipped 2025-03-14T05:04:16.2904857Z Running distributed tests for the test backend with env init_method 2025-03-14T05:04:16.2905373Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:04:16.2906681Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:04:16.282772] 2025-03-14T05:04:20.7936132Z 2025-03-14T05:04:20.7937298Z distributed/test_distributed_spawn 4/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.10_2e2778bf3ed098b4_.log 2025-03-14T05:04:20.7938403Z Running 0 items in this shard: 2025-03-14T05:04:20.7938653Z 2025-03-14T05:04:20.7940724Z Running distributed tests for the test backend with file init_method 2025-03-14T05:04:20.7942489Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:04:20.7947305Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:04:20.794541] 2025-03-14T05:04:25.2654117Z 2025-03-14T05:04:25.2655334Z distributed/test_distributed_spawn 4/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.10_3492154bb7e0a134_.log 2025-03-14T05:04:25.2656507Z Running 0 items in this shard: 2025-03-14T05:04:25.2659035Z 2025-03-14T05:04:25.2659364Z Running distributed tests for the nccl backend with env init_method 2025-03-14T05:04:25.2661249Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:04:25.2665536Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:04:25.266361] 2025-03-14T05:08:26.0002752Z 2025-03-14T05:08:26.0004051Z distributed/test_distributed_spawn 4/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.10_7f7b549a1ba83a50_.log 2025-03-14T05:08:26.0021074Z Running 27 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2025-03-14T05:08:26.0038091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2025-03-14T05:08:26.0039870Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2025-03-14T05:08:26.0041437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2025-03-14T05:08:26.0042681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2025-03-14T05:08:26.0044043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2025-03-14T05:08:26.0045528Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2025-03-14T05:08:26.0046804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2025-03-14T05:08:26.0048048Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2025-03-14T05:08:26.0049299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2025-03-14T05:08:26.0050555Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2025-03-14T05:08:26.0051890Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2025-03-14T05:08:26.0053205Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2025-03-14T05:08:26.0054591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2025-03-14T05:08:26.0055897Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2025-03-14T05:08:26.0057255Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2025-03-14T05:08:26.0058571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2025-03-14T05:08:26.0059929Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2025-03-14T05:08:26.0061106Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2025-03-14T05:08:26.0062229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2025-03-14T05:08:26.0063331Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2025-03-14T05:08:26.0064483Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2025-03-14T05:08:26.0065734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2025-03-14T05:08:26.0067007Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2025-03-14T05:08:26.0068294Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2025-03-14T05:08:26.0069702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2025-03-14T05:08:26.0071212Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2025-03-14T05:08:26.0072530Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2025-03-14T05:08:26.0073311Z 2025-03-14T05:08:26.0073599Z Running distributed tests for the nccl backend with file init_method 2025-03-14T05:08:26.0074156Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:08:26.0075560Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:08:26.002066] 2025-03-14T05:12:26.4539455Z 2025-03-14T05:12:26.4540640Z distributed/test_distributed_spawn 4/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.10_d880a192cb7a84f7_.log 2025-03-14T05:12:26.4556227Z Running 27 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2025-03-14T05:12:26.4572339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2025-03-14T05:12:26.4573966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2025-03-14T05:12:26.4575463Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2025-03-14T05:12:26.4576697Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2025-03-14T05:12:26.4578012Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2025-03-14T05:12:26.4579412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2025-03-14T05:12:26.4580725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2025-03-14T05:12:26.4582025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2025-03-14T05:12:26.4583171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2025-03-14T05:12:26.4584539Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2025-03-14T05:12:26.4585860Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2025-03-14T05:12:26.4587180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2025-03-14T05:12:26.4588490Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2025-03-14T05:12:26.4589797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2025-03-14T05:12:26.4591149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2025-03-14T05:12:26.4592458Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2025-03-14T05:12:26.4594081Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2025-03-14T05:12:26.4595292Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2025-03-14T05:12:26.4596444Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2025-03-14T05:12:26.4597578Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2025-03-14T05:12:26.4598763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2025-03-14T05:12:26.4600052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2025-03-14T05:12:26.4601350Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2025-03-14T05:12:26.4602605Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2025-03-14T05:12:26.4603847Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2025-03-14T05:12:26.4605286Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2025-03-14T05:12:26.4606643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2025-03-14T05:12:26.4607317Z 2025-03-14T05:12:26.4607573Z Running distributed tests for the gloo backend with env init_method 2025-03-14T05:12:26.4608081Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:12:26.4609391Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:12:26.455419] 2025-03-14T05:16:48.7956434Z 2025-03-14T05:16:48.7957653Z distributed/test_distributed_spawn 4/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.10_1208f7a57f541361_.log 2025-03-14T05:16:48.7974244Z Running 27 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2025-03-14T05:16:48.7989582Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2025-03-14T05:16:48.7991168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2025-03-14T05:16:48.7992592Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2025-03-14T05:16:48.7994058Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2025-03-14T05:16:48.7995388Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2025-03-14T05:16:48.7996792Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2025-03-14T05:16:48.7998113Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2025-03-14T05:16:48.7999323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2025-03-14T05:16:48.8000517Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2025-03-14T05:16:48.8001888Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2025-03-14T05:16:48.8003247Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2025-03-14T05:16:48.8004608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2025-03-14T05:16:48.8006143Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2025-03-14T05:16:48.8007416Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2025-03-14T05:16:48.8008723Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2025-03-14T05:16:48.8009998Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2025-03-14T05:16:48.8011251Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2025-03-14T05:16:48.8012447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2025-03-14T05:16:48.8013534Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2025-03-14T05:16:48.8014608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2025-03-14T05:16:48.8015724Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2025-03-14T05:16:48.8016925Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2025-03-14T05:16:48.8018154Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2025-03-14T05:16:48.8019336Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2025-03-14T05:16:48.8020510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2025-03-14T05:16:48.8021938Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2025-03-14T05:16:48.8023219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2025-03-14T05:16:48.8023908Z 2025-03-14T05:16:48.8024167Z Running distributed tests for the gloo backend with file init_method 2025-03-14T05:16:48.8024707Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:16:48.8026056Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:16:48.797308] 2025-03-14T05:21:11.0395860Z 2025-03-14T05:21:11.0397067Z distributed/test_distributed_spawn 4/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.10_df216ba9002c4717_.log 2025-03-14T05:21:11.0412690Z Running 27 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2025-03-14T05:21:11.0427864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2025-03-14T05:21:11.0429442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2025-03-14T05:21:11.0430864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2025-03-14T05:21:11.0432073Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2025-03-14T05:21:11.0433608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2025-03-14T05:21:11.0435020Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2025-03-14T05:21:11.0436335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2025-03-14T05:21:11.0437574Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2025-03-14T05:21:11.0438763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2025-03-14T05:21:11.0440055Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2025-03-14T05:21:11.0441478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2025-03-14T05:21:11.0442836Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2025-03-14T05:21:11.0444181Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2025-03-14T05:21:11.0445614Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2025-03-14T05:21:11.0446961Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2025-03-14T05:21:11.0448267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2025-03-14T05:21:11.0449554Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2025-03-14T05:21:11.0450807Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2025-03-14T05:21:11.0451923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2025-03-14T05:21:11.0453026Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2025-03-14T05:21:11.0454176Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2025-03-14T05:21:11.0455417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2025-03-14T05:21:11.0456691Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2025-03-14T05:21:11.0457914Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2025-03-14T05:21:11.0459129Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2025-03-14T05:21:11.0460425Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2025-03-14T05:21:11.0461701Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2025-03-14T05:21:11.0462396Z 2025-03-14T05:21:11.0462695Z Running distributed/test_distributed_spawn 7/10 ... [2025-03-14 05:21:11.041099] 2025-03-14T05:21:11.0463240Z /usr/bin/mpiexec 2025-03-14T05:21:11.0463620Z MPI not available -- MPI backend tests will be skipped 2025-03-14T05:21:11.0464201Z Running distributed tests for the test backend with env init_method 2025-03-14T05:21:11.0464725Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:21:11.0466070Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:21:11.042940] 2025-03-14T05:21:15.5132927Z 2025-03-14T05:21:15.5134130Z distributed/test_distributed_spawn 7/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.10_e2a4499c0bedac18_.log 2025-03-14T05:21:15.5135212Z Running 0 items in this shard: 2025-03-14T05:21:15.5135457Z 2025-03-14T05:21:15.5138111Z Running distributed tests for the test backend with file init_method 2025-03-14T05:21:15.5139892Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:21:15.5144406Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:21:15.514236] 2025-03-14T05:21:19.9900758Z 2025-03-14T05:21:19.9901987Z distributed/test_distributed_spawn 7/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.10_c70845217ecb2c87_.log 2025-03-14T05:21:19.9903084Z Running 0 items in this shard: 2025-03-14T05:21:19.9903327Z 2025-03-14T05:21:19.9905573Z Running distributed tests for the nccl backend with env init_method 2025-03-14T05:21:19.9907267Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:21:19.9911401Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:21:19.990952] 2025-03-14T05:25:55.8203402Z 2025-03-14T05:25:55.8204959Z distributed/test_distributed_spawn 7/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.10_e3ad0f310aa2c794_.log 2025-03-14T05:25:55.8221342Z Running 30 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2025-03-14T05:25:55.8238051Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2025-03-14T05:25:55.8239485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2025-03-14T05:25:55.8240772Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2025-03-14T05:25:55.8242021Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2025-03-14T05:25:55.8243381Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2025-03-14T05:25:55.8244925Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2025-03-14T05:25:55.8246201Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2025-03-14T05:25:55.8247383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2025-03-14T05:25:55.8248635Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-03-14T05:25:55.8249974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2025-03-14T05:25:55.8251336Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2025-03-14T05:25:55.8252579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2025-03-14T05:25:55.8253702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2025-03-14T05:25:55.8254815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2025-03-14T05:25:55.8255921Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2025-03-14T05:25:55.8257243Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2025-03-14T05:25:55.8258600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2025-03-14T05:25:55.8259849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2025-03-14T05:25:55.8261050Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2025-03-14T05:25:55.8262190Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2025-03-14T05:25:55.8263367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2025-03-14T05:25:55.8264594Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace 2025-03-14T05:25:55.8265870Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2025-03-14T05:25:55.8267136Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2025-03-14T05:25:55.8268445Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-03-14T05:25:55.8269815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2025-03-14T05:25:55.8271242Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-03-14T05:25:55.8272494Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2025-03-14T05:25:55.8273802Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-03-14T05:25:55.8274968Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2025-03-14T05:25:55.8275619Z 2025-03-14T05:25:55.8275886Z Running distributed tests for the nccl backend with file init_method 2025-03-14T05:25:55.8276554Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:25:55.8277949Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:25:55.822180] 2025-03-14T05:30:31.8933112Z 2025-03-14T05:30:31.8934317Z distributed/test_distributed_spawn 7/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.10_cac41c17512933c8_.log 2025-03-14T05:30:31.8950450Z Running 30 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2025-03-14T05:30:31.8967571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2025-03-14T05:30:31.8971112Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2025-03-14T05:30:31.8972596Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2025-03-14T05:30:31.8973860Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2025-03-14T05:30:31.8975220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2025-03-14T05:30:31.8976597Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2025-03-14T05:30:31.8977923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2025-03-14T05:30:31.8979144Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2025-03-14T05:30:31.8980430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-03-14T05:30:31.8981928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2025-03-14T05:30:31.8983360Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2025-03-14T05:30:31.8984556Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2025-03-14T05:30:31.8985645Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2025-03-14T05:30:31.8986725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2025-03-14T05:30:31.8987804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2025-03-14T05:30:31.8989092Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2025-03-14T05:30:31.8990406Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2025-03-14T05:30:31.8991630Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2025-03-14T05:30:31.8992882Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2025-03-14T05:30:31.8994384Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2025-03-14T05:30:31.8995605Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2025-03-14T05:30:31.8996872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace 2025-03-14T05:30:31.8998109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2025-03-14T05:30:31.8999423Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2025-03-14T05:30:31.9000744Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-03-14T05:30:31.9001990Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2025-03-14T05:30:31.9003251Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-03-14T05:30:31.9004582Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2025-03-14T05:30:31.9006005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-03-14T05:30:31.9007220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2025-03-14T05:30:31.9007804Z 2025-03-14T05:30:31.9008061Z Running distributed tests for the gloo backend with env init_method 2025-03-14T05:30:31.9008541Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:30:31.9009781Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:30:31.894847] 2025-03-14T05:35:12.5351821Z 2025-03-14T05:35:12.5353179Z distributed/test_distributed_spawn 7/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.10_6abc9d64bda0e7a9_.log 2025-03-14T05:35:12.5370795Z Running 30 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2025-03-14T05:35:12.5389235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2025-03-14T05:35:12.5390631Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2025-03-14T05:35:12.5391896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2025-03-14T05:35:12.5393379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2025-03-14T05:35:12.5394759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2025-03-14T05:35:12.5396135Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2025-03-14T05:35:12.5397455Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2025-03-14T05:35:12.5398831Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2025-03-14T05:35:12.5400165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-03-14T05:35:12.5401468Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2025-03-14T05:35:12.5402792Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2025-03-14T05:35:12.5403999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2025-03-14T05:35:12.5405188Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2025-03-14T05:35:12.5406213Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2025-03-14T05:35:12.5407235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2025-03-14T05:35:12.5408570Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2025-03-14T05:35:12.5409815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2025-03-14T05:35:12.5410968Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2025-03-14T05:35:12.5412074Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2025-03-14T05:35:12.5413306Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2025-03-14T05:35:12.5414455Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2025-03-14T05:35:12.5415655Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace 2025-03-14T05:35:12.5416837Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2025-03-14T05:35:12.5418140Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2025-03-14T05:35:12.5419483Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-03-14T05:35:12.5420594Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2025-03-14T05:35:12.5421709Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-03-14T05:35:12.5422815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2025-03-14T05:35:12.5423913Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-03-14T05:35:12.5424942Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2025-03-14T05:35:12.5425531Z 2025-03-14T05:35:12.5425773Z Running distributed tests for the gloo backend with file init_method 2025-03-14T05:35:12.5426259Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:35:12.5427492Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:35:12.536732] 2025-03-14T05:39:52.8759889Z 2025-03-14T05:39:52.8761105Z distributed/test_distributed_spawn 7/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.10_b01615d06992488a_.log 2025-03-14T05:39:52.8778493Z Running 30 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2025-03-14T05:39:52.8795309Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2025-03-14T05:39:52.8796749Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2025-03-14T05:39:52.8798035Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2025-03-14T05:39:52.8799289Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2025-03-14T05:39:52.8800653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2025-03-14T05:39:52.8802039Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2025-03-14T05:39:52.8803346Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2025-03-14T05:39:52.8804579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2025-03-14T05:39:52.8805966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-03-14T05:39:52.8807317Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2025-03-14T05:39:52.8808674Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2025-03-14T05:39:52.8809916Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2025-03-14T05:39:52.8811155Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2025-03-14T05:39:52.8812282Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2025-03-14T05:39:52.8813414Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2025-03-14T05:39:52.8814731Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2025-03-14T05:39:52.8816086Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2025-03-14T05:39:52.8817332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2025-03-14T05:39:52.8818536Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2025-03-14T05:39:52.8819734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2025-03-14T05:39:52.8820920Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2025-03-14T05:39:52.8822173Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace 2025-03-14T05:39:52.8823374Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2025-03-14T05:39:52.8824642Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2025-03-14T05:39:52.8825934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-03-14T05:39:52.8827143Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2025-03-14T05:39:52.8828357Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-03-14T05:39:52.8829564Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2025-03-14T05:39:52.8830748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-03-14T05:39:52.8831880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2025-03-14T05:39:52.8832511Z 2025-03-14T05:39:52.8832883Z Running distributed/test_distributed_spawn 10/10 ... [2025-03-14 05:39:52.877741] 2025-03-14T05:39:52.8833621Z /usr/bin/mpiexec 2025-03-14T05:39:52.8834009Z MPI not available -- MPI backend tests will be skipped 2025-03-14T05:39:52.8834606Z Running distributed tests for the test backend with env init_method 2025-03-14T05:39:52.8835145Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:39:52.8836539Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=10', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:39:52.879579] 2025-03-14T05:39:57.3567931Z 2025-03-14T05:39:57.3569157Z distributed/test_distributed_spawn 10/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_10.10_e9357b001ab413d6_.log 2025-03-14T05:39:57.3570737Z Running 0 items in this shard: 2025-03-14T05:39:57.3570991Z 2025-03-14T05:39:57.3575610Z Running distributed tests for the test backend with file init_method 2025-03-14T05:39:57.3577347Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:39:57.3581687Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=10', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:39:57.357973] 2025-03-14T05:40:01.8549408Z 2025-03-14T05:40:01.8550608Z distributed/test_distributed_spawn 10/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_10.10_705d3080ac729201_.log 2025-03-14T05:40:01.8551712Z Running 0 items in this shard: 2025-03-14T05:40:01.8551943Z 2025-03-14T05:40:01.8555165Z Running distributed tests for the nccl backend with env init_method 2025-03-14T05:40:01.8556958Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:40:01.8561245Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=10', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:40:01.855930] 2025-03-14T05:43:46.4758183Z 2025-03-14T05:43:46.4759921Z distributed/test_distributed_spawn 10/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_10.10_2e9557be05776e88_.log 2025-03-14T05:43:46.4775117Z Running 24 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-03-14T05:43:46.4789648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2025-03-14T05:43:46.4791229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2025-03-14T05:43:46.4792513Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2025-03-14T05:43:46.4794092Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2025-03-14T05:43:46.4795296Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2025-03-14T05:43:46.4796521Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2025-03-14T05:43:46.4797816Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2025-03-14T05:43:46.4799444Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2025-03-14T05:43:46.4801003Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2025-03-14T05:43:46.4802314Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2025-03-14T05:43:46.4803570Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2025-03-14T05:43:46.4804727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2025-03-14T05:43:46.4805800Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2025-03-14T05:43:46.4807031Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2025-03-14T05:43:46.4808320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2025-03-14T05:43:46.4809553Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2025-03-14T05:43:46.4810923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2025-03-14T05:43:46.4812301Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-03-14T05:43:46.4813478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2025-03-14T05:43:46.4814634Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2025-03-14T05:43:46.4815674Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2025-03-14T05:43:46.4816712Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2025-03-14T05:43:46.4817829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-03-14T05:43:46.4818994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-03-14T05:43:46.4819607Z 2025-03-14T05:43:46.4819913Z Running distributed tests for the nccl backend with file init_method 2025-03-14T05:43:46.4820411Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:43:46.4821657Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=10', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:43:46.477720] 2025-03-14T05:47:30.6315309Z 2025-03-14T05:47:30.6316498Z distributed/test_distributed_spawn 10/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_10.10_36d9f47aec036e6a_.log 2025-03-14T05:47:30.6331124Z Running 24 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-03-14T05:47:30.6344940Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2025-03-14T05:47:30.6346400Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2025-03-14T05:47:30.6347726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2025-03-14T05:47:30.6349063Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2025-03-14T05:47:30.6350403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2025-03-14T05:47:30.6351586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2025-03-14T05:47:30.6352913Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2025-03-14T05:47:30.6354515Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2025-03-14T05:47:30.6356032Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2025-03-14T05:47:30.6357426Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2025-03-14T05:47:30.6358885Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2025-03-14T05:47:30.6360114Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2025-03-14T05:47:30.6361315Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2025-03-14T05:47:30.6362623Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2025-03-14T05:47:30.6363999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2025-03-14T05:47:30.6365424Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2025-03-14T05:47:30.6366827Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2025-03-14T05:47:30.6368244Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-03-14T05:47:30.6369663Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2025-03-14T05:47:30.6371067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2025-03-14T05:47:30.6372237Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2025-03-14T05:47:30.6373404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2025-03-14T05:47:30.6374673Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-03-14T05:47:30.6375985Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-03-14T05:47:30.6376682Z 2025-03-14T05:47:30.6376956Z Running distributed tests for the gloo backend with env init_method 2025-03-14T05:47:30.6377506Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:47:30.6378901Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=10', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:47:30.633155] 2025-03-14T05:51:29.5374179Z 2025-03-14T05:51:29.5375756Z distributed/test_distributed_spawn 10/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_10.10_f78ab49fec7a5991_.log 2025-03-14T05:51:29.5390302Z Running 24 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-03-14T05:51:29.5405027Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2025-03-14T05:51:29.5406496Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2025-03-14T05:51:29.5407824Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2025-03-14T05:51:29.5409167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2025-03-14T05:51:29.5410335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2025-03-14T05:51:29.5411523Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2025-03-14T05:51:29.5412773Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2025-03-14T05:51:29.5414186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2025-03-14T05:51:29.5415672Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2025-03-14T05:51:29.5417090Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2025-03-14T05:51:29.5418391Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2025-03-14T05:51:29.5419576Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2025-03-14T05:51:29.5420672Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2025-03-14T05:51:29.5421939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2025-03-14T05:51:29.5423276Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2025-03-14T05:51:29.5424545Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2025-03-14T05:51:29.5425945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2025-03-14T05:51:29.5427420Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-03-14T05:51:29.5428626Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2025-03-14T05:51:29.5429721Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2025-03-14T05:51:29.5430855Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2025-03-14T05:51:29.5431981Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2025-03-14T05:51:29.5433438Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-03-14T05:51:29.5434754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-03-14T05:51:29.5435451Z 2025-03-14T05:51:29.5435719Z Running distributed tests for the gloo backend with file init_method 2025-03-14T05:51:29.5436269Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:51:29.5437664Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=10', '--num-shards=10', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:51:29.539179] 2025-03-14T05:55:28.4478364Z 2025-03-14T05:55:28.4479784Z distributed/test_distributed_spawn 10/10 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_10.10_aedef97445abce8f_.log 2025-03-14T05:55:28.4494387Z Running 24 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-03-14T05:55:28.4508474Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2025-03-14T05:55:28.4509942Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2025-03-14T05:55:28.4511288Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2025-03-14T05:55:28.4512644Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2025-03-14T05:55:28.4514113Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2025-03-14T05:55:28.4515341Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2025-03-14T05:55:28.4516635Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2025-03-14T05:55:28.4518071Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2025-03-14T05:55:28.4519603Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2025-03-14T05:55:28.4520997Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2025-03-14T05:55:28.4522334Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2025-03-14T05:55:28.4523560Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2025-03-14T05:55:28.4524802Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2025-03-14T05:55:28.4526070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2025-03-14T05:55:28.4527575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2025-03-14T05:55:28.4528815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2025-03-14T05:55:28.4530181Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2025-03-14T05:55:28.4531562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-03-14T05:55:28.4532734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2025-03-14T05:55:28.4533795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2025-03-14T05:55:28.4534899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2025-03-14T05:55:28.4536175Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2025-03-14T05:55:28.4537461Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-03-14T05:55:28.4538725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-03-14T05:55:28.4539383Z 2025-03-14T05:55:28.4539684Z Running distributed/fsdp/test_fsdp_core 3/3 ... [2025-03-14 05:55:28.449420] 2025-03-14T05:55:28.4540346Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T05:55:28.4541569Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=3', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 05:55:28.449771] 2025-03-14T06:02:04.2851547Z 2025-03-14T06:02:04.2852689Z distributed/fsdp/test_fsdp_core 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_3.3_1e13e702cce053f8_.log 2025-03-14T06:02:04.2868643Z Running 21 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_True_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_True_mixed_precision_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestNoGradCUDA::test_transformer_no_grad_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestAutogradCUDA::test_unshard_params_as_tensors_cuda 2025-03-14T06:02:04.2882310Z 2025-03-14T06:02:04.2882681Z Running distributed/fsdp/test_distributed_checkpoint 1/1 ... [2025-03-14 06:02:04.287792] 2025-03-14T06:02:04.2883346Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:02:04.2885032Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:02:04.288170] 2025-03-14T06:02:09.2656114Z 2025-03-14T06:02:09.2657318Z distributed/fsdp/test_distributed_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_bde77c173878fd45_.log 2025-03-14T06:02:09.2659857Z Running 2 items in this shard: test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type0_cuda, test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type1_cuda 2025-03-14T06:02:09.2661519Z 2025-03-14T06:02:09.2661836Z Running distributed/fsdp/test_fsdp_misc 1/1 ... [2025-03-14 06:02:09.265580] 2025-03-14T06:02:09.2662390Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:02:09.2663676Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:02:09.265963] 2025-03-14T06:03:36.0554519Z 2025-03-14T06:03:36.0555872Z distributed/fsdp/test_fsdp_misc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_7bba5eef084b80f0_.log 2025-03-14T06:03:36.0571914Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_cpu_init_with_sync_module_states, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_init_stays_on_cpu, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_training, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_False, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_True, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_not_all_outputs_used_in_loss, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optim_overlap_no_use_orig_params_error, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optimizer_overlap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_zero2_eval_with_prefetch, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_cpu_gpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_device_id_auto_wrap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_cpu_offload, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_no_move_ignored_params_and_bufs, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_ignored_module_meta, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_namedtuple, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_same_model_across_ranks, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_unsupported_module_cls, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_homogeneous_attributes, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_module_device_mismatches_device_id, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_multigpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_no_params, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_training_device_mismatch_errors, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_unsafe_setattr, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_world_size_1_sharding_strategy_warning 2025-03-14T06:03:36.0586717Z 2025-03-14T06:03:36.0586990Z Running distributed/test_c10d_nccl 1/3 ... [2025-03-14 06:03:36.055803] 2025-03-14T06:03:36.0587526Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:03:36.0588823Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:03:36.056213] 2025-03-14T06:20:01.7214620Z 2025-03-14T06:20:01.7216037Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_04f54c1768469a48_.log 2025-03-14T06:20:01.7259708Z Running 83 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_error_detection_and_propagation, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_restart_pg_after_error, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2025-03-14T06:20:01.7299165Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2025-03-14T06:20:01.7300081Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2025-03-14T06:20:01.7301085Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs 2025-03-14T06:20:01.7302066Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg 2025-03-14T06:20:01.7303045Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered 2025-03-14T06:20:01.7304092Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False 2025-03-14T06:20:01.7305334Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True 2025-03-14T06:20:01.7306395Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group 2025-03-14T06:20:01.7307513Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl 2025-03-14T06:20:01.7308625Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check 2025-03-14T06:20:01.7309721Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout 2025-03-14T06:20:01.7310937Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16 2025-03-14T06:20:01.7311999Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2 2025-03-14T06:20:01.7313149Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True 2025-03-14T06:20:01.7314453Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init 2025-03-14T06:20:01.7315616Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2025-03-14T06:20:01.7316855Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2025-03-14T06:20:01.7318035Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2025-03-14T06:20:01.7319323Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2025-03-14T06:20:01.7320799Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2025-03-14T06:20:01.7322240Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2025-03-14T06:20:01.7323674Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2025-03-14T06:20:01.7324932Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params 2025-03-14T06:20:01.7326176Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2025-03-14T06:20:01.7327400Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2025-03-14T06:20:01.7328794Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2025-03-14T06:20:01.7330060Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2025-03-14T06:20:01.7331396Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2025-03-14T06:20:01.7332786Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2025-03-14T06:20:01.7333940Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2025-03-14T06:20:01.7334993Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2025-03-14T06:20:01.7336177Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2025-03-14T06:20:01.7337295Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2025-03-14T06:20:01.7338448Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2025-03-14T06:20:01.7339502Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2025-03-14T06:20:01.7340666Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2025-03-14T06:20:01.7342040Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2025-03-14T06:20:01.7343244Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2025-03-14T06:20:01.7344252Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2025-03-14T06:20:01.7345254Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2025-03-14T06:20:01.7346551Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2025-03-14T06:20:01.7347661Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2025-03-14T06:20:01.7348693Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2025-03-14T06:20:01.7349667Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq 2025-03-14T06:20:01.7350709Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2025-03-14T06:20:01.7351770Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_error_detection_and_propagation 2025-03-14T06:20:01.7352956Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2025-03-14T06:20:01.7354313Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2025-03-14T06:20:01.7355502Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2025-03-14T06:20:01.7356695Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier 2025-03-14T06:20:01.7357862Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_restart_pg_after_error 2025-03-14T06:20:01.7358964Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors 2025-03-14T06:20:01.7360012Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2025-03-14T06:20:01.7361024Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced 2025-03-14T06:20:01.7362091Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2025-03-14T06:20:01.7363187Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2025-03-14T06:20:01.7364257Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2025-03-14T06:20:01.7365324Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor 2025-03-14T06:20:01.7366407Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2025-03-14T06:20:01.7367787Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2 2025-03-14T06:20:01.7369545Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2025-03-14T06:20:01.7371102Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True 2025-03-14T06:20:01.7372386Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False 2025-03-14T06:20:01.7373878Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True 2025-03-14T06:20:01.7375104Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True 2025-03-14T06:20:01.7376205Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False 2025-03-14T06:20:01.7377381Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False 2025-03-14T06:20:01.7378554Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True 2025-03-14T06:20:01.7379777Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False 2025-03-14T06:20:01.7381138Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False 2025-03-14T06:20:01.7382761Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False 2025-03-14T06:20:01.7383963Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False 2025-03-14T06:20:01.7385000Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2025-03-14T06:20:01.7385875Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe 2025-03-14T06:20:01.7386851Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True 2025-03-14T06:20:01.7387799Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2025-03-14T06:20:01.7388794Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False 2025-03-14T06:20:01.7390003Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False 2025-03-14T06:20:01.7391202Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True 2025-03-14T06:20:01.7392392Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False 2025-03-14T06:20:01.7393859Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True 2025-03-14T06:20:01.7395064Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired 2025-03-14T06:20:01.7396270Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2025-03-14T06:20:01.7396999Z 2025-03-14T06:20:01.7397323Z Running distributed/test_c10d_functional_native 1/1 ... [2025-03-14 06:20:01.723356] 2025-03-14T06:20:01.7397937Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:20:01.7399279Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:20:01.723762] 2025-03-14T06:23:09.5310524Z 2025-03-14T06:23:09.5311752Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_5023abcb7f83d749_.log 2025-03-14T06:23:09.5326886Z Running 30 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_fixed_striding, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_wait_tensor, test/distributed/test_c10d_functional_native.py::PyWorkTest::test_collectives, test/distributed/test_c10d_functional_native.py::PyWorkTest::test_wait_tensor, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_non_contig_input, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag, test/distributed/test_c10d_functional_native.py::CompileTest::test_wait_tensor 2025-03-14T06:23:09.5340037Z 2025-03-14T06:23:09.5340294Z Running distributed/test_c10d_pypg 1/1 ... [2025-03-14 06:23:09.530977] 2025-03-14T06:23:09.5340815Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:23:09.5342011Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:23:09.531380] 2025-03-14T06:23:16.5627903Z 2025-03-14T06:23:16.5629650Z distributed/test_c10d_pypg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_pypg_1.1_75010ceb3efbbbb1_.log 2025-03-14T06:23:16.5658947Z Running 46 items in this shard: test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_no_init_sync, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_no_init_sync, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_abort_shutdown, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_attr_overrides 2025-03-14T06:23:16.5683009Z 2025-03-14T06:23:16.5683299Z Running distributed/test_c10d_logger 1/1 ... [2025-03-14 06:23:16.562960] 2025-03-14T06:23:16.5683864Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:23:16.5685361Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:23:16.563537] 2025-03-14T06:23:33.2716183Z 2025-03-14T06:23:33.2717438Z distributed/test_c10d_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_logger_1.1_c6885017d23a1e2f_.log 2025-03-14T06:23:33.2719217Z Running 2 items in this shard: test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_exception_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_get_or_create_logger 2025-03-14T06:23:33.2720248Z 2025-03-14T06:23:33.2722181Z Running distributed/test_c10d_object_collectives 1/1 ... [2025-03-14 06:23:33.271893] 2025-03-14T06:23:33.2722780Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:23:33.2726016Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:23:33.272254] 2025-03-14T06:25:09.2785811Z 2025-03-14T06:25:09.2787142Z distributed/test_c10d_object_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_object_collectives_1.1_8cd8a722fe67e290_.log 2025-03-14T06:25:09.2793249Z Running 9 items in this shard: test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_all_gather_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_broadcast_object_list_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_gather_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_scatter_object_list_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_send_recv_object_list_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_subpg_all_gather_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_subpg_broadcast_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_subpg_gather_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_subpg_scatter_object_cuda 2025-03-14T06:25:09.2798612Z 2025-03-14T06:25:09.2798889Z Running distributed/test_c10d_gloo 1/2 ... [2025-03-14 06:25:09.278378] 2025-03-14T06:25:09.2799425Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:25:09.2800833Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:25:09.278750] 2025-03-14T06:35:15.3479508Z 2025-03-14T06:35:15.3480770Z distributed/test_c10d_gloo 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_1.2_1713d75f1c19317b_.log 2025-03-14T06:35:15.3510596Z Running 64 items in this shard: test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2025-03-14T06:35:15.3540495Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo 2025-03-14T06:35:15.3541551Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async 2025-03-14T06:35:15.3542645Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks 2025-03-14T06:35:15.3543768Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced 2025-03-14T06:35:15.3544852Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress 2025-03-14T06:35:15.3545876Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda 2025-03-14T06:35:15.3546896Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics 2025-03-14T06:35:15.3547937Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda 2025-03-14T06:35:15.3548962Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks 2025-03-14T06:35:15.3550004Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics 2025-03-14T06:35:15.3551123Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda 2025-03-14T06:35:15.3552198Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress 2025-03-14T06:35:15.3553449Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait 2025-03-14T06:35:15.3554504Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics 2025-03-14T06:35:15.3555559Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda 2025-03-14T06:35:15.3556611Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress 2025-03-14T06:35:15.3557658Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda 2025-03-14T06:35:15.3558697Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors 2025-03-14T06:35:15.3559723Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda 2025-03-14T06:35:15.3560884Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor 2025-03-14T06:35:15.3561947Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics 2025-03-14T06:35:15.3562954Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress 2025-03-14T06:35:15.3563979Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda 2025-03-14T06:35:15.3565110Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics 2025-03-14T06:35:15.3566162Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda 2025-03-14T06:35:15.3567080Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress 2025-03-14T06:35:15.3568008Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all 2025-03-14T06:35:15.3568946Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout 2025-03-14T06:35:15.3570565Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics 2025-03-14T06:35:15.3571692Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda 2025-03-14T06:35:15.3572816Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output 2025-03-14T06:35:15.3574021Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2025-03-14T06:35:15.3575482Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2025-03-14T06:35:15.3576909Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2025-03-14T06:35:15.3578150Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2025-03-14T06:35:15.3579661Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2025-03-14T06:35:15.3581030Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2025-03-14T06:35:15.3582344Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2025-03-14T06:35:15.3583682Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2025-03-14T06:35:15.3585059Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2025-03-14T06:35:15.3586361Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2025-03-14T06:35:15.3587593Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo 2025-03-14T06:35:15.3588815Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2025-03-14T06:35:15.3590004Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2025-03-14T06:35:15.3591368Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2025-03-14T06:35:15.3592610Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module 2025-03-14T06:35:15.3594044Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view 2025-03-14T06:35:15.3595255Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output 2025-03-14T06:35:15.3596477Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor 2025-03-14T06:35:15.3597609Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients 2025-03-14T06:35:15.3598788Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2025-03-14T06:35:15.3600034Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2025-03-14T06:35:15.3601121Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward 2025-03-14T06:35:15.3602192Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket 2025-03-14T06:35:15.3603221Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket 2025-03-14T06:35:15.3604249Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu 2025-03-14T06:35:15.3605272Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda 2025-03-14T06:35:15.3606288Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership 2025-03-14T06:35:15.3607186Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo 2025-03-14T06:35:15.3608115Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group 2025-03-14T06:35:15.3609008Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex 2025-03-14T06:35:15.3610061Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced 2025-03-14T06:35:15.3611382Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2025-03-14T06:35:15.3612581Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2025-03-14T06:35:15.3613147Z 2025-03-14T06:35:15.3613421Z Running distributed/tensor/test_attention 1/1 ... [2025-03-14 06:35:15.349532] 2025-03-14T06:35:15.3613949Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:35:15.3615129Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_attention.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:35:15.349896] 2025-03-14T06:40:33.5757959Z 2025-03-14T06:40:33.5758994Z PRINTING LOG FILE of distributed/tensor/test_attention 1/1 (test/test-reports/distributed.tensor.test_attention_1.1_cecac96004c5a704_.log) 2025-03-14T06:40:33.5760543Z Test results will be stored in test-reports/python-pytest/distributed.tensor.test_attention/distributed.tensor.test_attention-687e075f249a8bec.xml 2025-03-14T06:40:33.5761518Z ============================= test session starts ============================== 2025-03-14T06:40:33.5762277Z platform linux -- Python 3.10.16, pytest-7.3.2, pluggy-1.5.0 -- /opt/conda/envs/py_3.10/bin/python 2025-03-14T06:40:33.5763269Z cachedir: .pytest_cache 2025-03-14T06:40:33.5763988Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-03-14T06:40:33.5764782Z rootdir: /var/lib/jenkins/workspace 2025-03-14T06:40:33.5765161Z configfile: pytest.ini 2025-03-14T06:40:33.5765911Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.1.0, typeguard-4.3.0 2025-03-14T06:40:33.5766715Z collecting ... collected 4 items 2025-03-14T06:40:33.5767143Z stepcurrent: Cannot find last run test, not skipping 2025-03-14T06:40:33.5769605Z Running 4 items in this shard: test/distributed/tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior, test/distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer, test/distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer, test/distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa 2025-03-14T06:40:33.5771931Z 2025-03-14T06:40:33.5772401Z distributed/tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior PASSED [5.5277s] [ 25%] 2025-03-14T06:40:33.5773917Z distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer SKIPPED [0.0003s] (Does not support flash attention) [ 50%] 2025-03-14T06:40:33.5775449Z distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer SKIPPED [0.0002s] (Does not support flash attention) [ 75%] 2025-03-14T06:40:33.5776869Z distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa malloc(): unsorted double linked list corrupted 2025-03-14T06:40:33.5778123Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Process 0 timed out with traceback: 2025-03-14T06:40:33.5779163Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5780269Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f35bb11d700 (most recent call first): 2025-03-14T06:40:33.5781447Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5782632Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5783663Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f35bbb1e700 (most recent call first): 2025-03-14T06:40:33.5784771Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5785657Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5786873Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f35bc51f700 (most recent call first): 2025-03-14T06:40:33.5788019Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5788934Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5789994Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f35f5bdc700 (most recent call first): 2025-03-14T06:40:33.5791135Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5792043Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5793213Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Current thread 0x00007f3605f38700 (most recent call first): 2025-03-14T06:40:33.5795186Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 697 in _event_listener 2025-03-14T06:40:33.5797185Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 953 in run 2025-03-14T06:40:33.5798821Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 1016 in _bootstrap_inner 2025-03-14T06:40:33.5800475Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 973 in _bootstrap 2025-03-14T06:40:33.5801715Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5802806Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f374322a4c0 (most recent call first): 2025-03-14T06:40:33.5804511Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/var/lib/jenkins/workspace/test/distributed/tensor/test_attention.py", line 198 in _test_ring_attention_sdpa 2025-03-14T06:40:33.5806767Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1003 in run_subtests 2025-03-14T06:40:33.5808932Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 383 in run_subtests 2025-03-14T06:40:33.5811091Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/var/lib/jenkins/workspace/test/distributed/tensor/test_attention.py", line 71 in test_ring_attention_sdpa 2025-03-14T06:40:33.5813122Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 407 in wrapper 2025-03-14T06:40:33.5815145Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1875 in wrapper 2025-03-14T06:40:33.5817090Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 204 in wrapper 2025-03-14T06:40:33.5819143Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3153 in wrapper 2025-03-14T06:40:33.5821141Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 607 in wrapper 2025-03-14T06:40:33.5823108Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 734 in run_test 2025-03-14T06:40:33.5825063Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 713 in _run 2025-03-14T06:40:33.5826952Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 108 in run 2025-03-14T06:40:33.5828555Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap 2025-03-14T06:40:33.5830098Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 129 in _main 2025-03-14T06:40:33.5831628Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main 2025-03-14T06:40:33.5833013Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "", line 1 in 2025-03-14T06:40:33.5834179Z E0314 06:40:24.876000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5835215Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Process 1 timed out with traceback: 2025-03-14T06:40:33.5836254Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5837411Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f95cef17700 (most recent call first): 2025-03-14T06:40:33.5838594Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5839656Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5840617Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f95d0319700 (most recent call first): 2025-03-14T06:40:33.5841655Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5842489Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5843455Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f95cf918700 (most recent call first): 2025-03-14T06:40:33.5844727Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5845617Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5846637Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f960afde700 (most recent call first): 2025-03-14T06:40:33.5847740Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5848815Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5850004Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Current thread 0x00007f961ad82700 (most recent call first): 2025-03-14T06:40:33.5851691Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 697 in _event_listener 2025-03-14T06:40:33.5853451Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 953 in run 2025-03-14T06:40:33.5855040Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 1016 in _bootstrap_inner 2025-03-14T06:40:33.5856512Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 973 in _bootstrap 2025-03-14T06:40:33.5857610Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5858633Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f97583e44c0 (most recent call first): 2025-03-14T06:40:33.5860002Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 756 in __call__ 2025-03-14T06:40:33.5861771Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/tensor/debug/_comm_mode.py", line 681 in __torch_dispatch__ 2025-03-14T06:40:33.5863836Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/tensor/experimental/_attention.py", line 501 in _templated_ring_attention 2025-03-14T06:40:33.5866063Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/tensor/experimental/_attention.py", line 234 in _scaled_dot_product_ring_efficient_attention 2025-03-14T06:40:33.5868278Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/tensor/experimental/_attention.py", line 580 in _sdpa_handler 2025-03-14T06:40:33.5870959Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/tensor/_dispatch.py", line 164 in dispatch 2025-03-14T06:40:33.5873103Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/tensor/_api.py", line 344 in __torch_dispatch__ 2025-03-14T06:40:33.5875076Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838 in _fn 2025-03-14T06:40:33.5876884Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_compile.py", line 51 in inner 2025-03-14T06:40:33.5878872Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/tensor/experimental/_attention.py", line 991 in inner_fn 2025-03-14T06:40:33.5880906Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/var/lib/jenkins/workspace/test/distributed/tensor/test_attention.py", line 95 in fn_eval 2025-03-14T06:40:33.5882860Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/var/lib/jenkins/workspace/test/distributed/tensor/test_attention.py", line 176 in _test_ring_attention_sdpa 2025-03-14T06:40:33.5885178Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1003 in run_subtests 2025-03-14T06:40:33.5887156Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 383 in run_subtests 2025-03-14T06:40:33.5889093Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/var/lib/jenkins/workspace/test/distributed/tensor/test_attention.py", line 71 in test_ring_attention_sdpa 2025-03-14T06:40:33.5891004Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 407 in wrapper 2025-03-14T06:40:33.5893049Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1875 in wrapper 2025-03-14T06:40:33.5894874Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 204 in wrapper 2025-03-14T06:40:33.5896702Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3153 in wrapper 2025-03-14T06:40:33.5898534Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 607 in wrapper 2025-03-14T06:40:33.5900642Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 734 in run_test 2025-03-14T06:40:33.5902597Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 713 in _run 2025-03-14T06:40:33.5904369Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 108 in run 2025-03-14T06:40:33.5906001Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap 2025-03-14T06:40:33.5907637Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 129 in _main 2025-03-14T06:40:33.5909271Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main 2025-03-14T06:40:33.5910670Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "", line 1 in 2025-03-14T06:40:33.5911648Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5912626Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Process 2 timed out with traceback: 2025-03-14T06:40:33.5913847Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5914941Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007fc41b849700 (most recent call first): 2025-03-14T06:40:33.5916126Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5917073Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5918164Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007fc41cc4b700 (most recent call first): 2025-03-14T06:40:33.5919345Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5920286Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5921369Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007fc41c24a700 (most recent call first): 2025-03-14T06:40:33.5922612Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5923553Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5924648Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007fc455fff700 (most recent call first): 2025-03-14T06:40:33.5926018Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5926854Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5928071Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Current thread 0x00007fc467f38700 (most recent call first): 2025-03-14T06:40:33.5929748Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 697 in _event_listener 2025-03-14T06:40:33.5931567Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 953 in run 2025-03-14T06:40:33.5933100Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 1016 in _bootstrap_inner 2025-03-14T06:40:33.5934652Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 973 in _bootstrap 2025-03-14T06:40:33.5935814Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5936841Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007fc5a52ae4c0 (most recent call first): 2025-03-14T06:40:33.5938441Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/var/lib/jenkins/workspace/test/distributed/tensor/test_attention.py", line 198 in _test_ring_attention_sdpa 2025-03-14T06:40:33.5940486Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1003 in run_subtests 2025-03-14T06:40:33.5942479Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 383 in run_subtests 2025-03-14T06:40:33.5944630Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/var/lib/jenkins/workspace/test/distributed/tensor/test_attention.py", line 71 in test_ring_attention_sdpa 2025-03-14T06:40:33.5946663Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 407 in wrapper 2025-03-14T06:40:33.5948676Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1875 in wrapper 2025-03-14T06:40:33.5950611Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 204 in wrapper 2025-03-14T06:40:33.5952545Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3153 in wrapper 2025-03-14T06:40:33.5954852Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 607 in wrapper 2025-03-14T06:40:33.5956946Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 734 in run_test 2025-03-14T06:40:33.5959020Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 713 in _run 2025-03-14T06:40:33.5960902Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 108 in run 2025-03-14T06:40:33.5962650Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap 2025-03-14T06:40:33.5964447Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 129 in _main 2025-03-14T06:40:33.5966218Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main 2025-03-14T06:40:33.5967525Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "", line 1 in 2025-03-14T06:40:33.5968444Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5969499Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Process 3 timed out with traceback: 2025-03-14T06:40:33.5970669Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5971739Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f1f2dcb3700 (most recent call first): 2025-03-14T06:40:33.5972877Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5973794Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5974850Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f1f2f0b5700 (most recent call first): 2025-03-14T06:40:33.5975982Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5976891Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5978104Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f1f2e6b4700 (most recent call first): 2025-03-14T06:40:33.5979243Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5980151Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5981207Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f1f65fff700 (most recent call first): 2025-03-14T06:40:33.5982341Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5983442Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5984441Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Current thread 0x00007f1f77f38700 (most recent call first): 2025-03-14T06:40:33.5986149Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 697 in _event_listener 2025-03-14T06:40:33.5987815Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 953 in run 2025-03-14T06:40:33.5989276Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 1016 in _bootstrap_inner 2025-03-14T06:40:33.5990748Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 973 in _bootstrap 2025-03-14T06:40:33.5991856Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.5992829Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] Thread 0x00007f20b58124c0 (most recent call first): 2025-03-14T06:40:33.5994789Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/var/lib/jenkins/workspace/test/distributed/tensor/test_attention.py", line 198 in _test_ring_attention_sdpa 2025-03-14T06:40:33.5996903Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1003 in run_subtests 2025-03-14T06:40:33.5999116Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 383 in run_subtests 2025-03-14T06:40:33.6001308Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/var/lib/jenkins/workspace/test/distributed/tensor/test_attention.py", line 71 in test_ring_attention_sdpa 2025-03-14T06:40:33.6003458Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 407 in wrapper 2025-03-14T06:40:33.6005723Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1875 in wrapper 2025-03-14T06:40:33.6007542Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 204 in wrapper 2025-03-14T06:40:33.6009363Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3153 in wrapper 2025-03-14T06:40:33.6011196Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 607 in wrapper 2025-03-14T06:40:33.6013033Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 734 in run_test 2025-03-14T06:40:33.6014865Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 713 in _run 2025-03-14T06:40:33.6016602Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 108 in run 2025-03-14T06:40:33.6018146Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap 2025-03-14T06:40:33.6019682Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 129 in _main 2025-03-14T06:40:33.6021212Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main 2025-03-14T06:40:33.6022519Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] File "", line 1 in 2025-03-14T06:40:33.6023440Z E0314 06:40:24.877000 162135 site-packages/torch/testing/_internal/common_distributed.py:791] 2025-03-14T06:40:33.6024021Z ('RERUN', {'yellow': True}) [300.0924s] [100%] 2025-03-14T06:40:33.6024694Z distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa PASSED [7.9316s] [100%] 2025-03-14T06:40:33.6025284Z 2025-03-14T06:40:33.6025429Z ==================================== RERUNS ==================================== 2025-03-14T06:40:33.6025943Z __________________ RingAttentionTest.test_ring_attention_sdpa __________________ 2025-03-14T06:40:33.6026440Z Traceback (most recent call last): 2025-03-14T06:40:33.6027168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 605, in wrapper 2025-03-14T06:40:33.6027909Z self._join_processes(fn) 2025-03-14T06:40:33.6028643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 845, in _join_processes 2025-03-14T06:40:33.6029432Z self._check_return_codes(elapsed_time) 2025-03-14T06:40:33.6030231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 899, in _check_return_codes 2025-03-14T06:40:33.6031015Z raise RuntimeError( 2025-03-14T06:40:33.6031468Z RuntimeError: Process 0 terminated or timed out after 300.0884771347046 seconds 2025-03-14T06:40:33.6032072Z ----------------------------- Captured stdout call ----------------------------- 2025-03-14T06:40:33.6032584Z Timing out after 300 seconds and killing subprocesses. 2025-03-14T06:40:33.6033918Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.test_attention/distributed.tensor.test_attention-687e075f249a8bec.xml - 2025-03-14T06:40:33.6035134Z ============== 2 passed, 2 skipped, 1 rerun in 313.58s (0:05:13) =============== 2025-03-14T06:40:33.6035495Z 2025-03-14T06:40:33.6036168Z FINISHED PRINTING LOG FILE of distributed/tensor/test_attention 1/1 (test/test-reports/distributed.tensor.test_attention_1.1_cecac96004c5a704_.log) 2025-03-14T06:40:33.6036961Z 2025-03-14T06:40:33.6037329Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2025-03-14 06:40:33.576282] 2025-03-14T06:40:33.6037951Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:40:33.6039302Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:40:33.576644] 2025-03-14T06:40:43.3669391Z 2025-03-14T06:40:43.3670886Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_b472f4ff64767c22_.log 2025-03-14T06:40:43.3672386Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2025-03-14T06:40:43.3673424Z 2025-03-14T06:40:43.3673737Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2025-03-14 06:40:43.366893] 2025-03-14T06:40:43.3674306Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:40:43.3675604Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:40:43.367320] 2025-03-14T06:40:49.2976175Z 2025-03-14T06:40:49.2977166Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_0fe9c845225922a4_.log 2025-03-14T06:40:49.2978624Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracingCUDA::test_symbolic_tracing_outputs_cuda 2025-03-14T06:40:49.2979295Z 2025-03-14T06:40:49.2979692Z Running distributed/tensor/test_experimental_ops 1/1 ... [2025-03-14 06:40:49.297687] 2025-03-14T06:40:49.2980347Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:40:49.2982781Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:40:49.298060] 2025-03-14T06:41:12.0213352Z 2025-03-14T06:41:12.0214659Z distributed/tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_experimental_ops_1.1_63313631dc501593_.log 2025-03-14T06:41:12.0216973Z Running 3 items in this shard: test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2025-03-14T06:41:12.0218369Z 2025-03-14T06:41:12.0218820Z Running distributed/tensor/experimental/test_register_sharding 1/1 ... [2025-03-14 06:41:12.021543] 2025-03-14T06:41:12.0219592Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:41:12.0222022Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/experimental/test_register_sharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:41:12.021924] 2025-03-14T06:41:31.0364686Z 2025-03-14T06:41:31.0366402Z distributed/tensor/experimental/test_register_sharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.experimental.test_register_sharding_1.1_e07afc1483098d43_.log 2025-03-14T06:41:31.0370147Z Running 2 items in this shard: test/distributed/tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_argmax, test/distributed/tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_softmax_fwd 2025-03-14T06:41:31.0372152Z 2025-03-14T06:41:31.0372456Z Running distributed/test_backends 1/1 ... [2025-03-14 06:41:31.036375] 2025-03-14T06:41:31.0373011Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:41:31.0374299Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_backends.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:41:31.036754] 2025-03-14T06:41:36.8156434Z 2025-03-14T06:41:36.8157698Z distributed/test_backends 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_backends_1.1_66b0ef58f46d37e6_.log 2025-03-14T06:41:36.8159757Z Running 2 items in this shard: test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_create_pg_cuda, test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_device_to_backend_mapping_cuda 2025-03-14T06:41:36.8160873Z 2025-03-14T06:41:36.8161537Z Running distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 ... [2025-03-14 06:41:36.815890] 2025-03-14T06:41:36.8162157Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:41:36.8164574Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/fsdp/test_fsdp_dsd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:41:36.816270] 2025-03-14T06:42:30.2336974Z 2025-03-14T06:42:30.2339008Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_e7fb4bbd490746d5_.log 2025-03-14T06:42:30.2348513Z Running 6 items in this shard: test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_cpu_offload_full_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_get_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp2_tp_and_load_with_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_tp_and_load_with_fsdp2_tp 2025-03-14T06:42:30.2353346Z 2025-03-14T06:42:30.2353866Z Running distributed/tensor/test_init 1/1 ... [2025-03-14 06:42:30.233892] 2025-03-14T06:42:30.2354449Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:42:30.2355748Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:42:30.234459] 2025-03-14T06:43:24.4387345Z 2025-03-14T06:43:24.4389046Z distributed/tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_init_1.1_ac30809b44654a27_.log 2025-03-14T06:43:24.4395641Z Running 7 items in this shard: test/distributed/tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh 2025-03-14T06:43:24.4399683Z 2025-03-14T06:43:24.4400097Z Running distributed/checkpoint/test_fsdp_optim_state 1/1 ... [2025-03-14 06:43:24.438753] 2025-03-14T06:43:24.4400844Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:43:24.4402413Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:43:24.439310] 2025-03-14T06:43:43.4516172Z 2025-03-14T06:43:43.4517648Z distributed/checkpoint/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_bd173762af82a151_.log 2025-03-14T06:43:43.4520521Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_False, test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_True 2025-03-14T06:43:43.4522332Z 2025-03-14T06:43:43.4523225Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2025-03-14 06:43:43.452041] 2025-03-14T06:43:43.4523847Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:43:43.4526242Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:43:43.452426] 2025-03-14T06:44:00.1595687Z 2025-03-14T06:44:00.1596989Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_3086fafbb295b4fd_.log 2025-03-14T06:44:00.1599174Z Running 2 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_assert_same_keys, test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2025-03-14T06:44:00.1600411Z 2025-03-14T06:44:00.1600730Z Running distributed/_tools/test_memory_tracker 1/1 ... [2025-03-14 06:44:00.159663] 2025-03-14T06:44:00.1601461Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:44:00.1603099Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:44:00.160059] 2025-03-14T06:44:05.4880024Z 2025-03-14T06:44:05.4881257Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_be482d49d172595f_.log 2025-03-14T06:44:05.4882722Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model 2025-03-14T06:44:05.4883332Z 2025-03-14T06:44:05.4884710Z Running distributed/test_collective_utils 1/1 ... [2025-03-14 06:44:05.488173] 2025-03-14T06:44:05.4885400Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:44:05.4887266Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:44:05.488549] 2025-03-14T06:44:09.5577364Z 2025-03-14T06:44:09.5578711Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_08c3f9d244e6b6e4_.log 2025-03-14T06:44:09.5579671Z 2025-03-14T06:44:09.5581650Z Running distributed/tensor/test_random_ops 1/1 ... [2025-03-14 06:44:09.558020] 2025-03-14T06:44:09.5582432Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:44:09.5585915Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_random_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:44:09.558407] 2025-03-14T06:45:39.5065523Z 2025-03-14T06:45:39.5066824Z distributed/tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_random_ops_1.1_09b22c5ff8f303d5_.log 2025-03-14T06:45:39.5074095Z Running 12 items in this shard: test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_fsdp_tp_model_meta_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_meta_tensor_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_tp_model_meta_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed_submesh, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_pipeline_parallel_manual_seed, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpsTest3D::test_hsdp_tp_model_meta_init 2025-03-14T06:45:39.5080712Z 2025-03-14T06:45:39.5081042Z Running distributed/pipelining/test_backward 1/1 ... [2025-03-14 06:45:39.506547] 2025-03-14T06:45:39.5081621Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:45:39.5082892Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:45:39.506945] 2025-03-14T06:45:45.6871863Z 2025-03-14T06:45:45.6873259Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_99de1fa652728e8e_.log 2025-03-14T06:45:45.6877085Z Running 4 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTestsCUDA::test_stage_backward_cuda, test/distributed/pipelining/test_backward.py::StageBackwardTestsCUDA::test_stage_backward_input_cuda, test/distributed/pipelining/test_backward.py::StageBackwardTestsCUDA::test_stage_backward_weight_cuda, test/distributed/pipelining/test_backward.py::StageBackwardTestsCUDA::test_stage_backward_weight_multiple_iters_cuda 2025-03-14T06:45:45.6879556Z 2025-03-14T06:45:45.6879885Z Running distributed/pipelining/test_pipe 1/1 ... [2025-03-14 06:45:45.687166] 2025-03-14T06:45:45.6880463Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:45:45.6881749Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:45:45.687571] 2025-03-14T06:45:50.7153759Z 2025-03-14T06:45:50.7155029Z distributed/pipelining/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_pipe_1.1_6997959cab6d6475_.log 2025-03-14T06:45:50.7157497Z Running 3 items in this shard: test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass0, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass1, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass2 2025-03-14T06:45:50.7158927Z 2025-03-14T06:45:50.7159228Z Running distributed/test_device_mesh 1/1 ... [2025-03-14 06:45:50.715234] 2025-03-14T06:45:50.7159781Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:45:50.7161415Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:45:50.715643] 2025-03-14T06:50:38.3693531Z 2025-03-14T06:50:38.3694704Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_5b875dc34bd2af4f_.log 2025-03-14T06:50:38.3715738Z Running 48 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTestGlooBackend::test_device_mesh_reuse_default_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_non_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_init_backend, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_global_pg, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_invalid_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_and_get_all_groups, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_raises_invalid_device_type, test/distributed/test_device_mesh.py::DeviceMeshTest::test_set_mesh_dim_group_options, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_parent_child_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape_2d, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape_3d, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_get_local_rank_3d, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_4d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d_noncontiguous_slicing, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_reconstruct_mesh_with_flatten_dim, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_all_submeshes, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_not_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_mesh_slice_fake_tensor_mode, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven 2025-03-14T06:50:38.3735826Z 2025-03-14T06:50:38.3736213Z Running distributed/tensor/experimental/test_tp_transform 1/1 ... [2025-03-14 06:50:38.369495] 2025-03-14T06:50:38.3736857Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:50:38.3738385Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/experimental/test_tp_transform.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:50:38.369872] 2025-03-14T06:51:06.7072638Z 2025-03-14T06:51:06.7075047Z distributed/tensor/experimental/test_tp_transform 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.experimental.test_tp_transform_1.1_264dd061635863c2_.log 2025-03-14T06:51:06.7080408Z Running 3 items in this shard: test/distributed/tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_e2e, test/distributed/tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_no_bias, test/distributed/tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_with_uncovered_op 2025-03-14T06:51:06.7083529Z 2025-03-14T06:51:06.7084032Z Running distributed/tensor/test_utils 1/1 ... [2025-03-14 06:51:06.707720] 2025-03-14T06:51:06.7084958Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:51:06.7089253Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:51:06.708309] 2025-03-14T06:52:17.3016522Z 2025-03-14T06:52:17.3018063Z distributed/tensor/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_utils_1.1_ef6812c1a5262015_.log 2025-03-14T06:52:17.3023687Z Running 10 items in this shard: test/distributed/tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_1D, test/distributed/tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_2D, test/distributed/tensor/test_utils.py::UtilTest::test_fsdp_tp_meta_compute, test/distributed/tensor/test_utils.py::UtilTest::test_hsdp_tp_meta_compute, test/distributed/tensor/test_utils.py::UtilTest::test_strided_sharding_assumption_in_meta_compute, test/distributed/tensor/test_utils.py::TestStridedSharding::test_1d_mesh_strided_sharding, test/distributed/tensor/test_utils.py::TestStridedSharding::test_2d_mesh_2d_tensor_strided_sharding, test/distributed/tensor/test_utils.py::TestStridedSharding::test_2d_mesh_strided_sharding, test/distributed/tensor/test_utils.py::Test2DStridedLocalShard::test_fsdp1_tp_2d_dtensor_local_shards_and_offsets, test/distributed/tensor/test_utils.py::Test2DStridedLocalShard::test_fsdp2_tp_2d_dtensor_local_shards_and_offsets 2025-03-14T06:52:17.3028421Z 2025-03-14T06:52:17.3028823Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2025-03-14 06:52:17.301775] 2025-03-14T06:52:17.3029491Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:52:17.3030860Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:52:17.302179] 2025-03-14T06:52:28.3946594Z 2025-03-14T06:52:28.3947990Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_9b1a1f4fb8bbd663_.log 2025-03-14T06:52:28.3950462Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_post_optim_event_overlap, test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap 2025-03-14T06:52:28.3951881Z 2025-03-14T06:52:28.3953026Z Running distributed/pipelining/test_stage 1/1 ... [2025-03-14 06:52:28.394654] 2025-03-14T06:52:28.3953806Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:52:28.3955134Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_stage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:52:28.395045] 2025-03-14T06:52:40.4908536Z 2025-03-14T06:52:40.4909947Z distributed/pipelining/test_stage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_stage_1.1_d72c690ccddfb8de_.log 2025-03-14T06:52:40.4913674Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0 2025-03-14T06:52:40.4919049Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0 2025-03-14T06:52:40.4921616Z 2025-03-14T06:52:40.4921994Z Running distributed/checkpoint/test_hsdp_checkpoint 1/1 ... [2025-03-14 06:52:40.490616] 2025-03-14T06:52:40.4922642Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:52:40.4924032Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_hsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:52:40.491000] 2025-03-14T06:53:14.4409521Z 2025-03-14T06:53:14.4410850Z distributed/checkpoint/test_hsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_0acc1cfa96819b4c_.log 2025-03-14T06:53:14.4414556Z Running 4 items in this shard: test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_True, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_True 2025-03-14T06:53:14.4417318Z 2025-03-14T06:53:14.4417667Z Running distributed/test_compute_comm_reordering 1/1 ... [2025-03-14 06:53:14.440922] 2025-03-14T06:53:14.4418282Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:53:14.4419607Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:53:14.441324] 2025-03-14T06:55:16.4611637Z 2025-03-14T06:55:16.4613177Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_731181cab7811b7d_.log 2025-03-14T06:55:16.4618555Z Running 7 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2025-03-14T06:55:16.4624028Z 2025-03-14T06:55:16.4624304Z Running distributed/test_nccl 1/1 ... [2025-03-14 06:55:16.461139] 2025-03-14T06:55:16.4624820Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:55:16.4626036Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:55:16.461517] 2025-03-14T06:55:24.0449974Z 2025-03-14T06:55:24.0451097Z distributed/test_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nccl_1.1_4cc3de62eab05fdb_.log 2025-03-14T06:55:24.0456629Z Running 12 items in this shard: test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float8_e4m3fn, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float8_e5m2, test/distributed/test_nccl.py::TestNCCLCUDA::test_collective_errors_cuda, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_unique_id_cuda 2025-03-14T06:55:24.0461076Z 2025-03-14T06:55:24.0461433Z Running distributed/checkpoint/test_fsdp_model_state 1/1 ... [2025-03-14 06:55:24.045005] 2025-03-14T06:55:24.0462058Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:55:24.0463399Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:55:24.045412] 2025-03-14T06:55:43.4576470Z 2025-03-14T06:55:43.4578146Z distributed/checkpoint/test_fsdp_model_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_1a609f2b1c8dfecd_.log 2025-03-14T06:55:43.4580551Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_no_resharding, test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_with_resharding 2025-03-14T06:55:43.4582590Z 2025-03-14T06:55:43.4582984Z Running distributed/checkpoint/test_dtensor_checkpoint 1/1 ... [2025-03-14 06:55:43.458048] 2025-03-14T06:55:43.4583628Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:55:43.4586547Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:55:43.458423] 2025-03-14T06:55:55.2023809Z 2025-03-14T06:55:55.2025359Z distributed/checkpoint/test_dtensor_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_d72c52315128d20f_.log 2025-03-14T06:55:55.2027082Z Running 1 items in this shard: test/distributed/checkpoint/test_dtensor_checkpoint.py::DTensorPlanner::test_distributed_tensor_planner 2025-03-14T06:55:55.2028301Z 2025-03-14T06:55:55.2028632Z Running distributed/checkpoint/test_planner 1/1 ... [2025-03-14 06:55:55.202355] 2025-03-14T06:55:55.2029216Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:55:55.2030530Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:55:55.202726] 2025-03-14T06:56:00.2297241Z 2025-03-14T06:56:00.2298693Z distributed/checkpoint/test_planner 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_planner_1.1_271fc8d04a72ebc3_.log 2025-03-14T06:56:00.2306354Z Running 13 items in this shard: test/distributed/checkpoint/test_planner.py::TestSavePlan::test_finish_plan_with_caching, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan_with_caching, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_resharding, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_world_size_diff_by_one, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_load_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan_with_caching, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_compare_save_plans, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_create_read_item_from_chunks, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_merge_delta_local_plans, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_load_different_sizes_throws, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_strict 2025-03-14T06:56:00.2312449Z 2025-03-14T06:56:00.2312939Z Running distributed/checkpoint/test_dtensor_resharding 1/1 ... [2025-03-14 06:56:00.229692] 2025-03-14T06:56:00.2313789Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:56:00.2315201Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_resharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:56:00.230058] 2025-03-14T06:57:13.1462906Z 2025-03-14T06:57:13.1464282Z distributed/checkpoint/test_dtensor_resharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_990931a7831f3158_.log 2025-03-14T06:57:13.1470664Z Running 7 items in this shard: test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change_extensions0, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change_extensions1, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change_extensions2, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard 2025-03-14T06:57:13.1475845Z 2025-03-14T06:57:13.1476188Z Running distributed/optim/test_named_optimizer 1/1 ... [2025-03-14 06:57:13.146438] 2025-03-14T06:57:13.1476803Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:57:13.1478152Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:57:13.146849] 2025-03-14T06:57:16.2401219Z 2025-03-14T06:57:16.2402546Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_b5aca0775e2c614d_.log 2025-03-14T06:57:16.2403556Z 2025-03-14T06:57:16.2406208Z Running distributed/tensor/test_matrix_ops 1/1 ... [2025-03-14 06:57:16.240459] 2025-03-14T06:57:16.2406875Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:57:16.2410654Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_matrix_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:57:16.240861] 2025-03-14T06:58:54.0156301Z 2025-03-14T06:58:54.0157557Z distributed/tensor/test_matrix_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_matrix_ops_1.1_27832d6bdeca10c8_.log 2025-03-14T06:58:54.0164217Z Running 13 items in this shard: test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_auto_redistribute, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_empty_operand, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_baddbmm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_bmm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_dtensor_mm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_matmul, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_mm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_dot_product_attention, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_mm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_tensordot_shampoo 2025-03-14T06:58:54.0170476Z 2025-03-14T06:58:54.0170876Z Running distributed/fsdp/test_fsdp_backward_prefetch 1/1 ... [2025-03-14 06:58:54.015837] 2025-03-14T06:58:54.0171523Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:58:54.0172897Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_backward_prefetch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:58:54.016258] 2025-03-14T06:59:06.0607483Z 2025-03-14T06:59:06.0608774Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_e83714b1c2e5c064_.log 2025-03-14T06:59:06.0610431Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_backward_prefetch.py::TestBackwardPrefetch::test_backward_prefetch 2025-03-14T06:59:06.0611106Z 2025-03-14T06:59:06.0614682Z Running distributed/checkpoint/test_utils 1/1 ... [2025-03-14 06:59:06.060922] 2025-03-14T06:59:06.0615386Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:59:06.0617084Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:59:06.061304] 2025-03-14T06:59:25.2770622Z 2025-03-14T06:59:25.2771999Z distributed/checkpoint/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_utils_1.1_e71461eaeb7a068a_.log 2025-03-14T06:59:25.2778983Z Running 13 items in this shard: test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_dcp_logger, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_flat_data, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_equals, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_hash, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_init_convert_offset, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_sharded_tensor_lookup, test/distributed/checkpoint/test_utils.py::TestReaderView::testAllRead, test/distributed/checkpoint/test_utils.py::TestReaderView::testLongRead, test/distributed/checkpoint/test_utils.py::TestReaderView::testLongReadinto, test/distributed/checkpoint/test_utils.py::TestReaderView::testShortRead, test/distributed/checkpoint/test_utils.py::TestReaderView::testShortReadinto, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_gather_object, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_scatter_object 2025-03-14T06:59:25.2784995Z 2025-03-14T06:59:25.2785377Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2025-03-14 06:59:25.277115] 2025-03-14T06:59:25.2786010Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T06:59:25.2787491Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 06:59:25.277501] 2025-03-14T07:01:30.7194702Z 2025-03-14T07:01:30.7196060Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_6e9c488205a6bcf6_.log 2025-03-14T07:01:30.7209266Z Running 16 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_loss_parallel, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_float64_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_0_attention_wv__layers_0_feed_forward_w1__layers_1_feed_forward_w2__layers_1_ffn_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_1_ffn_norm__norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_weight_tying 2025-03-14T07:01:30.7222041Z 2025-03-14T07:01:30.7222464Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2025-03-14 07:01:30.719754] 2025-03-14T07:01:30.7223130Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:01:30.7224586Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:01:30.720142] 2025-03-14T07:02:05.4762367Z 2025-03-14T07:02:05.4763768Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_adb77f0a783078dc_.log 2025-03-14T07:02:05.4767420Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2025-03-14T07:02:05.4769719Z 2025-03-14T07:02:05.4770319Z Running distributed/_composable/fsdp/test_fully_shard_init 1/1 ... [2025-03-14 07:02:05.476605] 2025-03-14T07:02:05.4771011Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:02:05.4772475Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:02:05.477011] 2025-03-14T07:02:17.6208501Z 2025-03-14T07:02:17.6209884Z distributed/_composable/fsdp/test_fully_shard_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_init_1.1_e32f372dd6e0eb52_.log 2025-03-14T07:02:17.6236019Z Running 40 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceTensor::test_move_states_to_device_tensor, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_invalid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_valid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMeshArg::test_2d_mesh_without_mesh_dim_names, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMeshArg::test_invalid_mesh_ndim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_duplicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_list_of_mlps, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested_fully_shard_and_replicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_single, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_list_of_mlps, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_nested_fully_shard, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_shared_params_and_buffers, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_duplicates, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_list_of_mlps, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_shared_params, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_raise_noncontiguous_parameter, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_raise_scalar_parameter, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_shard_tensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterDTensor::test_shard_dtensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_double_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_is_root, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_module_and_param_fqns, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_multi_module_root, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_reset_sharded_param_in_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_invalid_meta_device_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_1d_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_2d_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_rank0_broadcast_meta_device_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardProcessGroupInit::test_1d_process_group_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardProcessGroupInit::test_2d_process_group_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardHSDPBroadcast::test_hsdp_broadcast_across_replicas, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestHSDPWithCustomHook::test_custom_hook_custom_stream, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestHSDPWithCustomHook::test_custom_hsdp_all_reduce_hook, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_1d_transformer_shard_dim_neg1, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_1d_transformer_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_1d_uneven_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_2d_transformer_shard_diff_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_invalid_shard_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardOldImport::test_old_import_training 2025-03-14T07:02:17.6260823Z 2025-03-14T07:02:17.6261260Z Running distributed/fsdp/test_fsdp_optim_state 1/1 ... [2025-03-14 07:02:17.620705] 2025-03-14T07:02:17.6261813Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:02:17.6263008Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:02:17.621091] 2025-03-14T07:09:58.6196375Z 2025-03-14T07:09:58.6197654Z distributed/fsdp/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_e8d2fb904d823585_.log 2025-03-14T07:09:58.6251256Z Running 60 items in this shard: test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_compatible_with_trec, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_nested, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_keys, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_nested_invalid, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_interface_arguments, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_no_grad, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_input_warning, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_without_param_groups, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_names, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type0, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type1, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_state_dict_with_none_tensor_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_use_orig_params, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_empty_optimizer_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_no_shard 2025-03-14T07:09:58.6294999Z 2025-03-14T07:09:58.6295406Z Running distributed/fsdp/test_fsdp_uneven 1/1 ... [2025-03-14 07:09:58.622797] 2025-03-14T07:09:58.6295971Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:09:58.6297217Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:09:58.623195] 2025-03-14T07:10:14.4786282Z 2025-03-14T07:10:14.4787507Z distributed/fsdp/test_fsdp_uneven 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_acd21a682c1b04bb_.log 2025-03-14T07:10:14.4790398Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda 2025-03-14T07:10:14.4791073Z 2025-03-14T07:10:14.4791551Z Running distributed/_composable/test_composability/test_2d_composability 1/1 ... [2025-03-14 07:10:14.478407] 2025-03-14T07:10:14.4792301Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:10:14.4794025Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_composability/test_2d_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:10:14.478781] 2025-03-14T07:12:52.3997702Z 2025-03-14T07:12:52.3999344Z distributed/_composable/test_composability/test_2d_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_2d_composability_1.1_66b549208c08cd35_.log 2025-03-14T07:12:52.4014912Z Running 20 items in this shard: test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_2d_uneven_shard_raise_error, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_tp_with_fsdp_offloading, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_mlp, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer_checkpoint_resume, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DStateDict::test_fully_shard_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::Test2dFSDP1ParallelIntegration::test_2d_ddp_integration_functionality, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_default, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_not_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_fsdp_state_enable_extension, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_raise_invalid_tp_composition, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp1_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp_2d_extension 2025-03-14T07:12:52.4029296Z 2025-03-14T07:12:52.4029603Z Running distributed/fsdp/test_fsdp_exec_order 1/1 ... [2025-03-14 07:12:52.400255] 2025-03-14T07:12:52.4030185Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:12:52.4031436Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:12:52.400620] 2025-03-14T07:14:24.0996830Z 2025-03-14T07:14:24.0998054Z distributed/fsdp/test_fsdp_exec_order 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_add41dca532558d3_.log 2025-03-14T07:14:24.1004182Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_first_iter_order_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_first_iter_order_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_1_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_3_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_1_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_3_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_train_eval_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_train_eval_sharding_strategy1_cuda 2025-03-14T07:14:24.1009727Z 2025-03-14T07:14:24.1010055Z Running distributed/test_dynamo_distributed 1/1 ... [2025-03-14 07:14:24.100038] 2025-03-14T07:14:24.1010635Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:14:24.1011921Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:14:24.100399] 2025-03-14T07:24:09.2636029Z 2025-03-14T07:24:09.2637089Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_89dabd6e3944091b_.log 2025-03-14T07:24:09.2666193Z Running 58 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_ddp_optimizer_inductor_strides_dont_specialize, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_dim_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_scalar_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_type_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_no_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_get_pg_attr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_full_model_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_local_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2025-03-14T07:24:09.2694983Z 2025-03-14T07:24:09.2695366Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2025-03-14 07:24:09.263887] 2025-03-14T07:24:09.2696005Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:24:09.2697338Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:24:09.264256] 2025-03-14T07:25:01.7598330Z 2025-03-14T07:25:01.7600638Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_20d2e084dd03c39d_.log 2025-03-14T07:25:01.7608044Z Running 9 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions0, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions1, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions2, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithCaching::test_read_write_shard_tensor 2025-03-14T07:25:01.7615126Z 2025-03-14T07:25:01.7615485Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2025-03-14 07:25:01.760450] 2025-03-14T07:25:01.7616128Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:25:01.7617461Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:25:01.760820] 2025-03-14T07:27:22.9861553Z 2025-03-14T07:27:22.9863643Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_20f16d6dc20e5f3c_.log 2025-03-14T07:27:22.9887761Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_grad_scaling, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_inf_gradients_skip_optim_step, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_scaling_unscaling_sparse, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_sharded_grad_scaler_found_inf 2025-03-14T07:27:22.9903479Z 2025-03-14T07:27:22.9903910Z Running distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2025-03-14 07:27:22.986713] 2025-03-14T07:27:22.9904583Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:27:22.9905928Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:27:22.987317] 2025-03-14T07:28:17.8911163Z 2025-03-14T07:28:17.8913824Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_3358339f4e23f51d_.log 2025-03-14T07:28:17.8933758Z Running 16 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadRot13::test_read_write_tensor_and_blob_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadRot13::test_read_write_tensor_and_blob_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadZStandard::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadZStandard::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_2 2025-03-14T07:28:17.8945563Z 2025-03-14T07:28:17.8945874Z Running distributed/launcher/test_run 1/1 ... [2025-03-14 07:28:17.891755] 2025-03-14T07:28:17.8946422Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:28:17.8947647Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/launcher/test_run.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:28:17.892339] 2025-03-14T07:29:05.4952203Z 2025-03-14T07:29:05.4953642Z distributed/launcher/test_run 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.launcher.test_run_1.1_2a36a401f8b2bd2d_.log 2025-03-14T07:29:05.4967672Z Running 24 items in this shard: test/distributed/launcher/test_run.py::ElasticLaunchTest::test_capture_logs_using_default_logs_specs, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_init_method_env_with_torchelastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_init_method_tcp_with_torchelastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_not_torchelastic_launched, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_torchelastic_launched, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_torchelastic_launched_with_logs_spec_defined, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_agent_raise_exception, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_multiple_agents, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_worker_raise_exception, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_run_path, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_shutdown, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_standalone, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_bash, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_default_nproc, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_python, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_python_caffe2_bc, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_with_env_vars, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_logs_logs_spec_entrypoint_must_be_defined, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_min_max_nodes_parse, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_gpu_launch_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_auto_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_number_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_unknown_configurations 2025-03-14T07:29:05.4979741Z 2025-03-14T07:29:05.4980090Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2025-03-14 07:29:05.495881] 2025-03-14T07:29:05.4980715Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:29:05.4982045Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:29:05.496284] 2025-03-14T07:29:10.0724622Z 2025-03-14T07:29:10.0726029Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_2ea4316eade43f52_.log 2025-03-14T07:29:10.0727300Z Running 0 items in this shard: 2025-03-14T07:29:10.0727535Z 2025-03-14T07:29:10.0729479Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 ... [2025-03-14 07:29:10.072479] 2025-03-14T07:29:10.0730170Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:29:10.0731546Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:29:10.072854] 2025-03-14T07:29:50.0421764Z 2025-03-14T07:29:50.0423125Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_c10d84e8b0942997_.log 2025-03-14T07:29:50.0426704Z Running 5 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_clone, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_deep_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_detach, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_inplace_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_set_requires_grad 2025-03-14T07:29:50.0429640Z 2025-03-14T07:29:50.0430043Z Running distributed/_shard/sharded_optim/test_sharded_optim 1/1 ... [2025-03-14 07:29:50.042301] 2025-03-14T07:29:50.0430701Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:29:50.0432063Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:29:50.042672] 2025-03-14T07:30:08.8571447Z 2025-03-14T07:30:08.8572828Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_f2e9eedbf3fda571_.log 2025-03-14T07:30:08.8577093Z Running 2 items in this shard: test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_named_params_with_sharded_tensor, test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_sharded_optim 2025-03-14T07:30:08.8578526Z 2025-03-14T07:30:08.8578809Z Running distributed/test_launcher 1/1 ... [2025-03-14 07:30:08.857322] 2025-03-14T07:30:08.8579345Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:30:08.8580584Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_launcher.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:30:08.857719] 2025-03-14T07:30:14.4888387Z 2025-03-14T07:30:14.4889762Z distributed/test_launcher 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_launcher_1.1_325ca958c68f67b8_.log 2025-03-14T07:30:14.4891135Z Running 1 items in this shard: test/distributed/test_launcher.py::TestDistributedLaunch::test_launch_user_script 2025-03-14T07:30:14.4891762Z 2025-03-14T07:30:14.4892119Z Running distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2025-03-14 07:30:14.488876] 2025-03-14T07:30:14.4892733Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:30:14.4894641Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:30:14.489257] 2025-03-14T07:31:15.1065739Z 2025-03-14T07:31:15.1066967Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_ec8fc303519d7cb3_.log 2025-03-14T07:31:15.1070200Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda 2025-03-14T07:31:15.1072312Z 2025-03-14T07:31:15.1074026Z Running distributed/_shard/sharded_tensor/ops/test_init 1/1 ... [2025-03-14 07:31:15.107233] 2025-03-14T07:31:15.1074686Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:31:15.1078713Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:31:15.107669] 2025-03-14T07:31:51.9646458Z 2025-03-14T07:31:51.9647802Z distributed/_shard/sharded_tensor/ops/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_373d7a5d7289d088_.log 2025-03-14T07:31:51.9650773Z Running 3 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_kaiming_uniform, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_normal, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_uniform 2025-03-14T07:31:51.9652767Z 2025-03-14T07:31:51.9653052Z Running distributed/_shard/test_sharder 1/1 ... [2025-03-14 07:31:51.964685] 2025-03-14T07:31:51.9653608Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:31:51.9654893Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:31:51.965102] 2025-03-14T07:32:10.6781105Z 2025-03-14T07:32:10.6782391Z distributed/_shard/test_sharder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.test_sharder_1.1_1e5aa8ced0a1795c_.log 2025-03-14T07:32:10.6784168Z Running 2 items in this shard: test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder, test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder_errors 2025-03-14T07:32:10.6785174Z 2025-03-14T07:32:10.6786317Z Running distributed/fsdp/test_wrap 1/1 ... [2025-03-14 07:32:10.678524] 2025-03-14T07:32:10.6786938Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:32:10.6790932Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:32:10.678914] 2025-03-14T07:35:16.6237663Z 2025-03-14T07:35:16.6238858Z distributed/fsdp/test_wrap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_wrap_1.1_9306b2c7b3faa3f7_.log 2025-03-14T07:35:16.6267384Z Running 52 items in this shard: test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_bn_always_wrapped_individually, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_False, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_True, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_zero_argument, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_api, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap_include_children, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf_custom, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_custom_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_frozen_params, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy_callable, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_transformer_auto_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_disabled_outside_context, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_override_defaults, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method1, test/distributed/fsdp/test_wrap.py::TestWrapUtils::test_validate_frozen_params 2025-03-14T07:35:16.6295363Z 2025-03-14T07:35:16.6295747Z Running distributed/optim/test_apply_optimizer_in_backward 1/1 ... [2025-03-14 07:35:16.624256] 2025-03-14T07:35:16.6296381Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:35:16.6297693Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:35:16.624691] 2025-03-14T07:35:19.7181526Z 2025-03-14T07:35:19.7183046Z distributed/optim/test_apply_optimizer_in_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_a145ae4b1a3d8d56_.log 2025-03-14T07:35:19.7184196Z 2025-03-14T07:35:19.7186003Z Running distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2025-03-14 07:35:19.718453] 2025-03-14T07:35:19.7186663Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:35:19.7190381Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:35:19.718821] 2025-03-14T07:35:51.9146197Z 2025-03-14T07:35:51.9148657Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_58e258755e75201a_.log 2025-03-14T07:35:51.9156279Z Running 11 items in this shard: test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_check_overlapping, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_chunked_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_device_placement, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_enumerable_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunk_sharding_params, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunked_dim_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_split_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_infer_sharding_spec_from_shards_metadata, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_shard_tensor, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_tensor_ctor 2025-03-14T07:35:51.9162855Z 2025-03-14T07:35:51.9163208Z Running distributed/fsdp/test_fsdp_use_orig_params 1/1 ... [2025-03-14 07:35:51.914989] 2025-03-14T07:35:51.9163818Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:35:51.9165128Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:35:51.915374] 2025-03-14T07:44:04.0222278Z 2025-03-14T07:44:04.0223585Z distributed/fsdp/test_fsdp_use_orig_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_fdcdec8a8426b810_.log 2025-03-14T07:44:04.0241716Z Running 25 items in this shard: test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_trainability, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_fsdp_compile, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_multiple_optimizers, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsParamAccess::test_access_params_after_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_grad_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_no_reshard_and_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_param_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_between_fwd_and_bwd_for_no_reshard_raises, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_shape_mismatch, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsFQNs::test_named_parameters_in_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_correctness, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsInit::test_non_uniform_requires_grad, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cpu, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cuda 2025-03-14T07:44:04.0258288Z 2025-03-14T07:44:04.0258563Z Running distributed/test_c10d_ops_nccl 1/1 ... [2025-03-14 07:44:04.023094] 2025-03-14T07:44:04.0259099Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:44:04.0260330Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ops_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:44:04.023498] 2025-03-14T07:44:20.7287518Z 2025-03-14T07:44:20.7288715Z distributed/test_c10d_ops_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ops_nccl_1.1_4084e5ba55b71056_.log 2025-03-14T07:44:20.7300463Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2025-03-14T07:44:20.7322351Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2025-03-14T07:44:20.7344026Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2025-03-14T07:44:20.7366378Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2025-03-14T07:44:20.7377942Z 2025-03-14T07:44:20.7378285Z Running distributed/_tools/test_fake_collectives 1/1 ... [2025-03-14 07:44:20.728959] 2025-03-14T07:44:20.7378905Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:44:20.7380255Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_fake_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:44:20.729333] 2025-03-14T07:44:25.7074065Z 2025-03-14T07:44:25.7075395Z distributed/_tools/test_fake_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_fake_collectives_1.1_0ee0a27d47da00af_.log 2025-03-14T07:44:25.7077292Z Running 1 items in this shard: test/distributed/_tools/test_fake_collectives.py::TestFakeCollectives::test_collectives 2025-03-14T07:44:25.7077938Z 2025-03-14T07:44:25.7078290Z Running distributed/elastic/events/lib_test 1/1 ... [2025-03-14 07:44:25.707644] 2025-03-14T07:44:25.7079031Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:44:25.7082188Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/events/lib_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:44:25.708031] 2025-03-14T07:44:30.3848625Z 2025-03-14T07:44:30.3849938Z distributed/elastic/events/lib_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.events.lib_test_1.1_1193088999bba53b_.log 2025-03-14T07:44:30.3854609Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2025-03-14T07:44:30.3858582Z 2025-03-14T07:44:30.3858910Z Running distributed/elastic/metrics/api_test 1/1 ... [2025-03-14 07:44:30.384863] 2025-03-14T07:44:30.3859520Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:44:30.3860838Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:44:30.385239] 2025-03-14T07:44:34.9618124Z 2025-03-14T07:44:34.9619377Z distributed/elastic/metrics/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.metrics.api_test_1.1_501c26aa3a3a83fd_.log 2025-03-14T07:44:34.9621679Z Running 3 items in this shard: test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_get_metric_name, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_inheritance, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_profile 2025-03-14T07:44:34.9623469Z 2025-03-14T07:44:34.9623854Z Running distributed/elastic/multiprocessing/api_test 1/1 ... [2025-03-14 07:44:34.961829] 2025-03-14T07:44:34.9624496Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:44:34.9625876Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:44:34.962200] 2025-03-14T07:45:14.7303417Z 2025-03-14T07:45:14.7304773Z distributed/elastic/multiprocessing/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_d8bc2db5aaf0aedc_.log 2025-03-14T07:45:14.7320447Z Running 25 items in this shard: test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_get_failures, test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_is_failed, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_str_bad_input, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait_on_a_child_thread, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_void_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_wait_for_all_child_procs_to_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsFuncTest::test_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary_redirect_and_tee 2025-03-14T07:45:14.7334362Z 2025-03-14T07:45:14.7334733Z Running distributed/elastic/timer/local_timer_example 1/1 ... [2025-03-14 07:45:14.730357] 2025-03-14T07:45:14.7335371Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:45:14.7336720Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:45:14.730735] 2025-03-14T07:45:37.2023240Z 2025-03-14T07:45:37.2025545Z distributed/elastic/timer/local_timer_example 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_example_1.1_10945b9933129ac0_.log 2025-03-14T07:45:37.2027713Z Running 2 items in this shard: test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_example_start_method_spawn, test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_torch_mp_example 2025-03-14T07:45:37.2028957Z 2025-03-14T07:45:37.2032854Z Running distributed/elastic/timer/local_timer_test 1/1 ... [2025-03-14 07:45:37.202677] 2025-03-14T07:45:37.2033673Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:45:37.2035053Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:45:37.203048] 2025-03-14T07:45:47.6957179Z 2025-03-14T07:45:47.6958530Z distributed/elastic/timer/local_timer_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_test_1.1_d691d350873f01b0_.log 2025-03-14T07:45:47.6966741Z Running 14 items in this shard: test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_client_interaction, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_exception_propagation, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_get_timer_recursive, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_happy_path, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_no_client, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_timer, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_less_than_size, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_size, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_acquire_release, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_expired_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_valid_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_call_count, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_empty_queue 2025-03-14T07:45:47.6974123Z 2025-03-14T07:45:47.6974498Z Running distributed/elastic/utils/distributed_test 1/1 ... [2025-03-14 07:45:47.696036] 2025-03-14T07:45:47.6975131Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:45:47.6976576Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:45:47.696401] 2025-03-14T07:45:58.3448680Z 2025-03-14T07:45:58.3450019Z distributed/elastic/utils/distributed_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.distributed_test_1.1_d959277c8c24e587_.log 2025-03-14T07:45:58.3455547Z Running 8 items in this shard: test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_no_port_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_single_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_worker, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_with_libuv_support, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_worker 2025-03-14T07:45:58.3460443Z 2025-03-14T07:45:58.3460788Z Running distributed/elastic/utils/logging_test 1/1 ... [2025-03-14 07:45:58.345106] 2025-03-14T07:45:58.3461391Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:45:58.3462704Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:45:58.345479] 2025-03-14T07:46:03.0224984Z 2025-03-14T07:46:03.0226278Z distributed/elastic/utils/logging_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.logging_test_1.1_52d00981044b3469_.log 2025-03-14T07:46:03.0228247Z Running 2 items in this shard: test/distributed/elastic/utils/logging_test.py::LoggingTest::test_derive_module_name, test/distributed/elastic/utils/logging_test.py::LoggingTest::test_logger_name 2025-03-14T07:46:03.0229253Z 2025-03-14T07:46:03.0231342Z Running distributed/elastic/utils/util_test 1/1 ... [2025-03-14 07:46:03.022636] 2025-03-14T07:46:03.0231977Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:46:03.0233565Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:46:03.023009] 2025-03-14T07:46:07.8498525Z 2025-03-14T07:46:07.8500397Z distributed/elastic/utils/util_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.util_test_1.1_7f56f554d46c499e_.log 2025-03-14T07:46:07.8510419Z Running 12 items in this shard: test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_hash_store, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_operations, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_rank_tracing, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_0, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_n, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize_hash_store, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_custom_name, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_different, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_none 2025-03-14T07:46:07.8518905Z 2025-03-14T07:46:08.6575453Z Running test batch 'tests to run' cost 11227.44 seconds 2025-03-14T07:46:09.5160991Z 2025-03-14T07:46:09.5161464Z real 187m13.846s 2025-03-14T07:46:09.5161829Z user 450m20.792s 2025-03-14T07:46:09.5162138Z sys 196m24.751s 2025-03-14T07:46:09.5162438Z + assert_git_not_dirty 2025-03-14T07:46:09.5162863Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2025-03-14T07:46:09.5163353Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2025-03-14T07:46:09.5167016Z ++ git status --porcelain 2025-03-14T07:46:09.5168490Z ++ grep -v '?? third_party' 2025-03-14T07:46:12.6660056Z ++ true 2025-03-14T07:46:12.6660377Z + git_status= 2025-03-14T07:46:12.6660673Z + [[ -n '' ]] 2025-03-14T07:46:12.6661011Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2025-03-14T07:46:12.6661506Z + [[ 1 == 1 ]] 2025-03-14T07:46:12.6661815Z + echo 'Testing distributed C++ tests' 2025-03-14T07:46:12.6662210Z Testing distributed C++ tests 2025-03-14T07:46:12.6665605Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-03-14T07:46:12.6678762Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-03-14T07:46:12.6693242Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-03-14T07:46:12.6694045Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-03-14T07:46:12.6694710Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2025-03-14T07:46:12.7760955Z /var/lib/jenkins/workspace/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2025-03-14T07:46:12.7762065Z import pkg_resources 2025-03-14T07:46:18.0566489Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-03-14T07:46:18.1309261Z Ignoring disabled issues: [''] 2025-03-14T07:46:18.1498906Z Found test times from artifacts 2025-03-14T07:46:18.2251201Z Found test times from artifacts 2025-03-14T07:46:18.2275008Z Running all tests 2025-03-14T07:46:18.2276701Z Running parallel tests on 2 processes 2025-03-14T07:46:18.2277148Z Name: tests to run (est. time: 0.0min) 2025-03-14T07:46:18.2277572Z Serial tests (1): 2025-03-14T07:46:18.2277890Z cpp/FileStoreTest 1/1 2025-03-14T07:46:18.2278231Z Parallel tests (0): 2025-03-14T07:46:18.2278562Z Name: excluded (est. time: 0.0min) 2025-03-14T07:46:18.2278926Z Serial tests (0): 2025-03-14T07:46:18.2279228Z Parallel tests (0): 2025-03-14T07:46:18.2337697Z Running cpp/FileStoreTest 1/1 ... [2025-03-14 07:46:18.233220] 2025-03-14T07:46:18.2338272Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:46:18.2339743Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-5cd143f5be00104a.xml', '-x', '--reruns=2'] ... [2025-03-14 07:46:18.233768] 2025-03-14T07:46:21.5577391Z 2025-03-14T07:46:21.5578446Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_ea6c756c02b54124_.log 2025-03-14T07:46:21.5579279Z 2025-03-14T07:46:22.9393655Z Running test batch 'tests to run' cost 4.71 seconds 2025-03-14T07:46:23.7778147Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2025-03-14T07:46:23.8867215Z /var/lib/jenkins/workspace/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2025-03-14T07:46:23.8868243Z import pkg_resources 2025-03-14T07:46:29.1679528Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-03-14T07:46:29.1867975Z Found test times from artifacts 2025-03-14T07:46:29.2620598Z Found test times from artifacts 2025-03-14T07:46:29.2643328Z Running all tests 2025-03-14T07:46:29.2645802Z Running parallel tests on 2 processes 2025-03-14T07:46:29.2646624Z Name: tests to run (est. time: 0.0min) 2025-03-14T07:46:29.2647072Z Serial tests (1): 2025-03-14T07:46:29.2647387Z cpp/HashStoreTest 1/1 2025-03-14T07:46:29.2647988Z Parallel tests (0): 2025-03-14T07:46:29.2648318Z Name: excluded (est. time: 0.0min) 2025-03-14T07:46:29.2648673Z Serial tests (0): 2025-03-14T07:46:29.2648969Z Parallel tests (0): 2025-03-14T07:46:29.2704546Z Running cpp/HashStoreTest 1/1 ... [2025-03-14 07:46:29.269935] 2025-03-14T07:46:29.2705174Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:46:29.2706620Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-6ba91ba33eb388ac.xml', '-x', '--reruns=2'] ... [2025-03-14 07:46:29.270433] 2025-03-14T07:46:32.2934674Z 2025-03-14T07:46:32.2935684Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_d321ea168b05577d_.log 2025-03-14T07:46:32.2936451Z 2025-03-14T07:46:34.0042236Z Running test batch 'tests to run' cost 4.74 seconds 2025-03-14T07:46:34.8393835Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2025-03-14T07:46:34.9471503Z /var/lib/jenkins/workspace/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2025-03-14T07:46:34.9472569Z import pkg_resources 2025-03-14T07:46:40.2317119Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-03-14T07:46:40.2505621Z Found test times from artifacts 2025-03-14T07:46:40.3261710Z Found test times from artifacts 2025-03-14T07:46:40.3288273Z Running all tests 2025-03-14T07:46:40.3288670Z Running parallel tests on 2 processes 2025-03-14T07:46:40.3289080Z Name: tests to run (est. time: 0.0min) 2025-03-14T07:46:40.3289453Z Serial tests (1): 2025-03-14T07:46:40.3289762Z cpp/TCPStoreTest 1/1 2025-03-14T07:46:40.3290090Z Parallel tests (0): 2025-03-14T07:46:40.3290415Z Name: excluded (est. time: 0.0min) 2025-03-14T07:46:40.3290806Z Serial tests (0): 2025-03-14T07:46:40.3291105Z Parallel tests (0): 2025-03-14T07:46:40.3346963Z Running cpp/TCPStoreTest 1/1 ... [2025-03-14 07:46:40.334175] 2025-03-14T07:46:40.3347515Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:46:40.3349350Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-51f49187f74c33f3.xml', '-x', '--reruns=2'] ... [2025-03-14 07:46:40.334662] 2025-03-14T07:46:54.9528607Z 2025-03-14T07:46:54.9529682Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_e57b4d8c21b7ef55_.log 2025-03-14T07:46:54.9530437Z 2025-03-14T07:46:55.7535235Z Running test batch 'tests to run' cost 15.42 seconds 2025-03-14T07:46:56.5861273Z + echo 'Testing multi-GPU linalg tests' 2025-03-14T07:46:56.5861805Z Testing multi-GPU linalg tests 2025-03-14T07:46:56.5862443Z + python test/run_test.py -i test_linalg.py -k test_matmul_offline_mgpu_tunable --verbose 2025-03-14T07:46:56.6937536Z /var/lib/jenkins/workspace/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2025-03-14T07:46:56.6938600Z import pkg_resources 2025-03-14T07:47:01.9806693Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-03-14T07:47:01.9995513Z Found test times from artifacts 2025-03-14T07:47:02.0749928Z Found test times from artifacts 2025-03-14T07:47:02.0773147Z Running all tests 2025-03-14T07:47:02.0776382Z Running parallel tests on 2 processes 2025-03-14T07:47:02.0776860Z Name: tests to run (est. time: 0.01min) 2025-03-14T07:47:02.0777250Z Serial tests (1): 2025-03-14T07:47:02.0777561Z test_linalg 1/1 2025-03-14T07:47:02.0777859Z Parallel tests (0): 2025-03-14T07:47:02.0778169Z Name: excluded (est. time: 0.0min) 2025-03-14T07:47:02.0778846Z Serial tests (0): 2025-03-14T07:47:02.0779141Z Parallel tests (0): 2025-03-14T07:47:02.0834750Z Running test_linalg 1/1 ... [2025-03-14 07:47:02.082825] 2025-03-14T07:47:02.0835271Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:47:02.0836654Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_linalg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_matmul_offline_mgpu_tunable', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-03-14 07:47:02.083189] 2025-03-14T07:47:08.9156762Z 2025-03-14T07:47:08.9157939Z test_linalg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_linalg_1.1_a8d367a0c85dc6f3_.log 2025-03-14T07:47:08.9159160Z Running 1 items in this shard: test/test_linalg.py::TestLinalgCUDA::test_matmul_offline_mgpu_tunableop_cuda_float32 2025-03-14T07:47:08.9159888Z 2025-03-14T07:47:09.7062029Z Running test batch 'tests to run' cost 7.63 seconds 2025-03-14T07:47:10.5623761Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2025-03-14T07:47:10.5624620Z ++ command -v mpiexec 2025-03-14T07:47:10.5626314Z + MPIEXEC=/usr/bin/mpiexec 2025-03-14T07:47:10.5626661Z + [[ -n /usr/bin/mpiexec ]] 2025-03-14T07:47:10.5627686Z + MPICMD='/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2025-03-14T07:47:10.5628767Z + eval '/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2025-03-14T07:47:10.5629780Z ++ /usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest 2025-03-14T07:47:12.7159282Z Test successful 2025-03-14T07:47:12.7217066Z Test successful 2025-03-14T07:47:12.8809857Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupGlooTest 2025-03-14T07:47:12.9877255Z /var/lib/jenkins/workspace/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2025-03-14T07:47:12.9878346Z import pkg_resources 2025-03-14T07:47:18.3357421Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-03-14T07:47:18.3548492Z Found test times from artifacts 2025-03-14T07:47:18.4299856Z Found test times from artifacts 2025-03-14T07:47:18.4323403Z Running all tests 2025-03-14T07:47:18.4325808Z Running parallel tests on 2 processes 2025-03-14T07:47:18.4326552Z Name: tests to run (est. time: 0.0min) 2025-03-14T07:47:18.4326979Z Serial tests (1): 2025-03-14T07:47:18.4327308Z cpp/ProcessGroupGlooTest 1/1 2025-03-14T07:47:18.4327677Z Parallel tests (0): 2025-03-14T07:47:18.4328027Z Name: excluded (est. time: 0.0min) 2025-03-14T07:47:18.4328370Z Serial tests (0): 2025-03-14T07:47:18.4328676Z Parallel tests (0): 2025-03-14T07:47:18.4380464Z Running cpp/ProcessGroupGlooTest 1/1 ... [2025-03-14 07:47:18.437840] 2025-03-14T07:47:18.4381073Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:47:18.4385284Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-ee4efc38a445a361.xml', '-x', '--reruns=2'] ... [2025-03-14 07:47:18.438331] 2025-03-14T07:47:26.7258219Z 2025-03-14T07:47:26.7259359Z cpp/ProcessGroupGlooTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupGlooTest_1.1_2aac4f1dfd96abaa_.log 2025-03-14T07:47:27.5314230Z 2025-03-14T07:47:27.5314798Z Running test batch 'tests to run' cost 9.1 seconds 2025-03-14T07:47:28.3715589Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLTest 2025-03-14T07:47:28.4795452Z /var/lib/jenkins/workspace/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2025-03-14T07:47:28.4796792Z import pkg_resources 2025-03-14T07:47:33.7912833Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-03-14T07:47:33.8101057Z Found test times from artifacts 2025-03-14T07:47:33.8861531Z Found test times from artifacts 2025-03-14T07:47:33.8888254Z Running all tests 2025-03-14T07:47:33.8888689Z Running parallel tests on 2 processes 2025-03-14T07:47:33.8889348Z Name: tests to run (est. time: 0.0min) 2025-03-14T07:47:33.8889724Z Serial tests (1): 2025-03-14T07:47:33.8890061Z cpp/ProcessGroupNCCLTest 1/1 2025-03-14T07:47:33.8890412Z Parallel tests (0): 2025-03-14T07:47:33.8890737Z Name: excluded (est. time: 0.0min) 2025-03-14T07:47:33.8891092Z Serial tests (0): 2025-03-14T07:47:33.8891385Z Parallel tests (0): 2025-03-14T07:47:33.8949297Z Running cpp/ProcessGroupNCCLTest 1/1 ... [2025-03-14 07:47:33.894425] 2025-03-14T07:47:33.8949874Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:47:33.8951387Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-d83301cd0686e283.xml', '-x', '--reruns=2'] ... [2025-03-14 07:47:33.894906] 2025-03-14T07:47:54.2115892Z 2025-03-14T07:47:54.2117388Z cpp/ProcessGroupNCCLTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLTest_1.1_7b66f4037c17f7ef_.log 2025-03-14T07:47:54.2118359Z 2025-03-14T07:47:55.0139836Z Running test batch 'tests to run' cost 21.12 seconds 2025-03-14T07:47:55.8795206Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLErrorsTest 2025-03-14T07:47:55.9870813Z /var/lib/jenkins/workspace/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2025-03-14T07:47:55.9871827Z import pkg_resources 2025-03-14T07:48:01.3227596Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-03-14T07:48:01.3414698Z Found test times from artifacts 2025-03-14T07:48:01.4161177Z Found test times from artifacts 2025-03-14T07:48:01.4191153Z Running all tests 2025-03-14T07:48:01.4191650Z Running parallel tests on 2 processes 2025-03-14T07:48:01.4192080Z Name: tests to run (est. time: 0.0min) 2025-03-14T07:48:01.4192454Z Serial tests (1): 2025-03-14T07:48:01.4192788Z cpp/ProcessGroupNCCLErrorsTest 1/1 2025-03-14T07:48:01.4193301Z Parallel tests (0): 2025-03-14T07:48:01.4193810Z Name: excluded (est. time: 0.0min) 2025-03-14T07:48:01.4194161Z Serial tests (0): 2025-03-14T07:48:01.4194468Z Parallel tests (0): 2025-03-14T07:48:01.4244406Z Running cpp/ProcessGroupNCCLErrorsTest 1/1 ... [2025-03-14 07:48:01.424232] 2025-03-14T07:48:01.4245049Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:48:01.4249851Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-d703c6fe075629f5.xml', '-x', '--reruns=2'] ... [2025-03-14 07:48:01.424751] 2025-03-14T07:49:06.8185144Z 2025-03-14T07:49:06.8186364Z cpp/ProcessGroupNCCLErrorsTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_b24dd91790bb736c_.log 2025-03-14T07:49:06.8187295Z 2025-03-14T07:49:07.6171048Z Running test batch 'tests to run' cost 66.2 seconds 2025-03-14T07:49:08.4373185Z + [[ 1 == 1 ]] 2025-03-14T07:49:08.4373556Z + test_rpc 2025-03-14T07:49:08.4373855Z + echo 'Testing RPC C++ tests' 2025-03-14T07:49:08.4374242Z Testing RPC C++ tests 2025-03-14T07:49:08.4377710Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-03-14T07:49:08.4394099Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-03-14T07:49:08.4406561Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-03-14T07:49:08.4407210Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2025-03-14T07:49:08.5483452Z /var/lib/jenkins/workspace/test/run_test.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2025-03-14T07:49:08.5484533Z import pkg_resources 2025-03-14T07:49:13.8763789Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-03-14T07:49:13.8953412Z Found test times from artifacts 2025-03-14T07:49:13.9709304Z Found test times from artifacts 2025-03-14T07:49:13.9736503Z Running all tests 2025-03-14T07:49:13.9736886Z Running parallel tests on 2 processes 2025-03-14T07:49:13.9737302Z Name: tests to run (est. time: 0.0min) 2025-03-14T07:49:13.9737681Z Serial tests (1): 2025-03-14T07:49:13.9737993Z cpp/test_cpp_rpc 1/1 2025-03-14T07:49:13.9738322Z Parallel tests (0): 2025-03-14T07:49:13.9738635Z Name: excluded (est. time: 0.0min) 2025-03-14T07:49:13.9738999Z Serial tests (0): 2025-03-14T07:49:13.9739300Z Parallel tests (0): 2025-03-14T07:49:13.9793300Z Running cpp/test_cpp_rpc 1/1 ... [2025-03-14 07:49:13.978840] 2025-03-14T07:49:13.9794050Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-03-14T07:49:13.9796250Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-4b3315072aafddcd.xml', '-x', '--reruns=2'] ... [2025-03-14 07:49:13.979358] 2025-03-14T07:49:20.6613848Z 2025-03-14T07:49:20.6615360Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_468aabb211f7f9db_.log 2025-03-14T07:49:20.6616712Z 2025-03-14T07:49:21.4596191Z Running test batch 'tests to run' cost 7.49 seconds 2025-03-14T07:49:22.2920326Z + cleanup_workspace 2025-03-14T07:49:22.2921084Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-03-14T07:49:22.2922059Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-03-14T07:49:22.2922875Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-03-14T07:49:22.2923463Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-03-14T07:49:22.2924138Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-03-14T07:49:22.2924885Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-03-14T07:49:22.2925585Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-03-14T07:49:22.8540362Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-03-14T07:49:22.8540944Z with: 2025-03-14T07:49:22.8541257Z benchmark-results-dir: test/test-reports 2025-03-14T07:49:22.8541653Z dry-run: false 2025-03-14T07:49:22.8541945Z schema-version: v3 2025-03-14T07:49:22.8542478Z github-token: *** 2025-03-14T07:49:22.8542767Z env: 2025-03-14T07:49:22.8543039Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:22.8543453Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:22.8544338Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:22.8545001Z ##[endgroup] 2025-03-14T07:49:22.8578344Z ##[group]Run set -eux 2025-03-14T07:49:22.8578682Z set -eux 2025-03-14T07:49:22.8579027Z python3 -mpip install boto3==1.35.33 2025-03-14T07:49:22.8588904Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:22.8589326Z env: 2025-03-14T07:49:22.8589624Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:22.8589986Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:22.8590586Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:22.8591124Z ##[endgroup] 2025-03-14T07:49:22.8622189Z + python3 -mpip install boto3==1.35.33 2025-03-14T07:49:23.1125713Z Defaulting to user installation because normal site-packages is not writeable 2025-03-14T07:49:24.2579762Z Collecting boto3==1.35.33 2025-03-14T07:49:24.2602465Z Using cached boto3-1.35.33-py3-none-any.whl (139 kB) 2025-03-14T07:49:24.2669769Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (1.35.99) 2025-03-14T07:49:24.2673002Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.4) 2025-03-14T07:49:24.2679420Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-03-14T07:49:24.2743948Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-03-14T07:49:24.2749172Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.2) 2025-03-14T07:49:24.2795066Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-03-14T07:49:24.4514595Z Installing collected packages: boto3 2025-03-14T07:49:24.4515060Z Attempting uninstall: boto3 2025-03-14T07:49:24.4516028Z Found existing installation: boto3 1.35.42 2025-03-14T07:49:24.4625013Z Uninstalling boto3-1.35.42: 2025-03-14T07:49:24.4639586Z Successfully uninstalled boto3-1.35.42 2025-03-14T07:49:24.5250839Z Successfully installed boto3-1.35.33 2025-03-14T07:49:24.5926204Z ##[group]Run set -eux 2025-03-14T07:49:24.5926535Z set -eux 2025-03-14T07:49:24.5926812Z  2025-03-14T07:49:24.5927089Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-03-14T07:49:24.5927489Z  echo "Missing github-token input" 2025-03-14T07:49:24.5927853Z  exit 1 2025-03-14T07:49:24.5928109Z fi 2025-03-14T07:49:24.5933699Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:24.5934143Z env: 2025-03-14T07:49:24.5934405Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:24.5934781Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:24.5935384Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:24.5936178Z GITHUB_TOKEN: *** 2025-03-14T07:49:24.5936465Z ##[endgroup] 2025-03-14T07:49:24.5962527Z + [[ -z *** ]] 2025-03-14T07:49:24.6019240Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-03-14T07:49:24.6019927Z with: 2025-03-14T07:49:24.6020322Z github-token: *** 2025-03-14T07:49:24.6020608Z env: 2025-03-14T07:49:24.6020867Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:24.6021260Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:24.6021868Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:24.6022415Z ##[endgroup] 2025-03-14T07:49:24.6055058Z ##[group]Run set -eux 2025-03-14T07:49:24.6055363Z set -eux 2025-03-14T07:49:24.6055762Z  2025-03-14T07:49:24.6056291Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-03-14T07:49:24.6061903Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:24.6062341Z env: 2025-03-14T07:49:24.6062591Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:24.6062968Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:24.6063587Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:24.6064347Z GITHUB_TOKEN: *** 2025-03-14T07:49:24.6064795Z ##[endgroup] 2025-03-14T07:49:24.6089740Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 13849515290 i-0f8cc103977fc9a05 2025-03-14T07:49:26.6989437Z setting job-id=38755323799 2025-03-14T07:49:26.6990226Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T07:49:26.7126079Z ##[group]Run set -eux 2025-03-14T07:49:26.7126580Z set -eux 2025-03-14T07:49:26.7126879Z  2025-03-14T07:49:26.7127354Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-03-14T07:49:26.7127940Z  --schema-version "${SCHEMA_VERSION}" \ 2025-03-14T07:49:26.7128372Z  --repo "${REPO}" \ 2025-03-14T07:49:26.7128745Z  --head-branch "${HEAD_BRANCH}" \ 2025-03-14T07:49:26.7129166Z  --head-sha "${HEAD_SHA}" \ 2025-03-14T07:49:26.7129555Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-03-14T07:49:26.7129997Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-03-14T07:49:26.7130385Z  --job-id "${JOB_ID}" \ 2025-03-14T07:49:26.7130745Z  --job-name "${JOB_NAME}" 2025-03-14T07:49:26.7137185Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:26.7137663Z env: 2025-03-14T07:49:26.7137952Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:26.7138384Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:26.7139063Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:26.7139654Z SCHEMA_VERSION: v3 2025-03-14T07:49:26.7140000Z REPO: pytorch/pytorch 2025-03-14T07:49:26.7140341Z HEAD_BRANCH: refs/heads/main 2025-03-14T07:49:26.7140738Z HEAD_SHA: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T07:49:26.7141293Z WORKFLOW_RUN_ID: 13849515290 2025-03-14T07:49:26.7141612Z RUN_ATTEMPT: 1 2025-03-14T07:49:26.7141891Z JOB_ID: 38755323799 2025-03-14T07:49:26.7142453Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T07:49:26.7143073Z ##[endgroup] 2025-03-14T07:49:26.7172098Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha aed0b7a742a2d7b7901790622829cbd2135049a4 --workflow-id 13849515290 --run-attempt 1 --job-id 38755323799 --job-name 'linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu)' 2025-03-14T07:49:26.7537589Z ##[group]Run set -eux 2025-03-14T07:49:26.7537907Z set -eux 2025-03-14T07:49:26.7538180Z  2025-03-14T07:49:26.7538478Z # TODO (huydhn): Implement this part 2025-03-14T07:49:26.7539073Z echo "runners=[]" >> "${GITHUB_OUTPUT}" 2025-03-14T07:49:26.7544874Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:26.7545308Z env: 2025-03-14T07:49:26.7545555Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:26.7545938Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:26.7546552Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:26.7547094Z ##[endgroup] 2025-03-14T07:49:26.7573408Z + echo 'runners=[]' 2025-03-14T07:49:26.7603871Z ##[group]Run set -eux 2025-03-14T07:49:26.7604214Z set -eux 2025-03-14T07:49:26.7604513Z  2025-03-14T07:49:26.7604842Z # TODO (huydhn): Implement this part 2025-03-14T07:49:26.7605447Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-03-14T07:49:26.7611298Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:26.7611732Z env: 2025-03-14T07:49:26.7611986Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:26.7612375Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:26.7612982Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:26.7613523Z ##[endgroup] 2025-03-14T07:49:26.7637159Z + echo 'dependencies={}' 2025-03-14T07:49:26.7668919Z ##[group]Run set -eux 2025-03-14T07:49:26.7669459Z set -eux 2025-03-14T07:49:26.7669922Z  2025-03-14T07:49:26.7670422Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-03-14T07:49:26.7670999Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-03-14T07:49:26.7671619Z  # We don't want the job to fail if the directory doesn't exist 2025-03-14T07:49:26.7672105Z  exit 0 2025-03-14T07:49:26.7672392Z fi 2025-03-14T07:49:26.7687213Z  2025-03-14T07:49:26.7687663Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-03-14T07:49:26.7688234Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-03-14T07:49:26.7688878Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-03-14T07:49:26.7689379Z  --metadata "${BENCHMARK_METADATA}" \ 2025-03-14T07:49:26.7689789Z  --runners "${RUNNER_INFO}" \ 2025-03-14T07:49:26.7690198Z  --dependencies "${DEPENDENCIES}" \ 2025-03-14T07:49:26.7690588Z  --dry-run 2025-03-14T07:49:26.7690874Z else 2025-03-14T07:49:26.7691311Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-03-14T07:49:26.7691946Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-03-14T07:49:26.7692439Z  --metadata "${BENCHMARK_METADATA}" \ 2025-03-14T07:49:26.7692856Z  --runners "${RUNNER_INFO}" \ 2025-03-14T07:49:26.7693266Z  --dependencies "${DEPENDENCIES}" 2025-03-14T07:49:26.7693641Z fi 2025-03-14T07:49:26.7699024Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:26.7699478Z env: 2025-03-14T07:49:26.7699735Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:26.7700129Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:26.7700879Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:26.7701444Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-03-14T07:49:26.7701790Z DRY_RUN: false 2025-03-14T07:49:26.7703253Z BENCHMARK_METADATA: {"timestamp": 1741938566, "schema_version": "v3", "name": "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "aed0b7a742a2d7b7901790622829cbd2135049a4", "workflow_id": 13849515290, "run_attempt": 1, "job_id": 38755323799} 2025-03-14T07:49:26.7704784Z RUNNER_INFO: [] 2025-03-14T07:49:26.7705049Z DEPENDENCIES: {} 2025-03-14T07:49:26.7705311Z ##[endgroup] 2025-03-14T07:49:26.7728614Z + [[ ! -d test/test-reports ]] 2025-03-14T07:49:26.7730036Z + [[ false == \t\r\u\e ]] 2025-03-14T07:49:26.7732861Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1741938566, "schema_version": "v3", "name": "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "aed0b7a742a2d7b7901790622829cbd2135049a4", "workflow_id": 13849515290, "run_attempt": 1, "job_id": 38755323799}' --runners '[]' --dependencies '{}' 2025-03-14T07:49:26.9458047Z ##[group]Run cat test/**/*_toprint.log || true 2025-03-14T07:49:26.9458473Z cat test/**/*_toprint.log || true 2025-03-14T07:49:26.9464046Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:26.9464479Z env: 2025-03-14T07:49:26.9464738Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:26.9465130Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:26.9465756Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:26.9466316Z ##[endgroup] 2025-03-14T07:49:26.9543726Z cat: 'test/**/*_toprint.log': No such file or directory 2025-03-14T07:49:26.9587744Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-03-14T07:49:26.9588146Z kill "$MONITOR_SCRIPT_PID" 2025-03-14T07:49:26.9593547Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:26.9594052Z env: 2025-03-14T07:49:26.9594317Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:26.9594739Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:26.9595412Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:26.9596030Z MONITOR_SCRIPT_PID: 750389 2025-03-14T07:49:26.9596377Z ##[endgroup] 2025-03-14T07:49:26.9756359Z Prepare all required actions 2025-03-14T07:49:26.9756856Z Getting action download info 2025-03-14T07:49:27.0764518Z Download action repository 'actions/upload-artifact@v4' (SHA:4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1) 2025-03-14T07:49:27.5965448Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-03-14T07:49:27.5965997Z with: 2025-03-14T07:49:27.5966469Z file-suffix: test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799 2025-03-14T07:49:27.5967051Z s3-bucket: gha-artifacts 2025-03-14T07:49:27.5967368Z env: 2025-03-14T07:49:27.5967616Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:27.5968030Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:27.5968673Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:27.5969439Z ##[endgroup] 2025-03-14T07:49:27.6018810Z ##[group]Run # Remove any previous test jsons if they exist 2025-03-14T07:49:27.6019405Z # Remove any previous test jsons if they exist 2025-03-14T07:49:27.6019847Z rm -f test-jsons-*.zip 2025-03-14T07:49:27.6020353Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-03-14T07:49:27.6026260Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:27.6026727Z env: 2025-03-14T07:49:27.6026991Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:27.6027384Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:27.6028017Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:27.6028893Z FILE_SUFFIX: test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799 2025-03-14T07:49:27.6029431Z ##[endgroup] 2025-03-14T07:49:27.6105382Z adding: test/test-reports/td_exclusions-91e109434725a21e065f.json (deflated 85%) 2025-03-14T07:49:27.6106458Z adding: test/test-reports/td_exclusions-96baa5bfeff8bc2e565c.json (deflated 14%) 2025-03-14T07:49:27.6107243Z adding: test/test-reports/td_exclusions-db099b3157b859547f24.json (deflated 15%) 2025-03-14T07:49:27.6108002Z adding: test/test-reports/td_exclusions-67065ba074c2b50baedc.json (deflated 18%) 2025-03-14T07:49:27.6108759Z adding: test/test-reports/td_exclusions-551c83a592aad2af859a.json (deflated 14%) 2025-03-14T07:49:27.6109514Z adding: test/test-reports/td_exclusions-7300888f7e094d389abb.json (deflated 20%) 2025-03-14T07:49:27.6110298Z adding: test/test-reports/td_exclusions-4cc9230f9c74dcb2c144.json (deflated 12%) 2025-03-14T07:49:27.6111126Z adding: test/test-reports/td_exclusions-2fbe1c7d90589b24b249.json (deflated 12%) 2025-03-14T07:49:27.6112114Z adding: test/test-reports/td_exclusions-53f4268839b281abedce.json (deflated 11%) 2025-03-14T07:49:27.6139206Z ##[group]Run # Remove any previous test reports if they exist 2025-03-14T07:49:27.6139754Z # Remove any previous test reports if they exist 2025-03-14T07:49:27.6140207Z rm -f test-reports-*.zip 2025-03-14T07:49:27.6140780Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-03-14T07:49:27.6146473Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:27.6146921Z env: 2025-03-14T07:49:27.6147174Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:27.6147573Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:27.6148207Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:27.6148979Z FILE_SUFFIX: test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799 2025-03-14T07:49:27.6149535Z ##[endgroup] 2025-03-14T07:49:27.6258142Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-366bbe7f1ec1cd26.xml (deflated 39%) 2025-03-14T07:49:27.6259699Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1799d58a4c05797a.xml (deflated 38%) 2025-03-14T07:49:27.6261422Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7482f4713bb1113c.xml (deflated 37%) 2025-03-14T07:49:27.6262916Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eba49b8e11187469.xml (deflated 38%) 2025-03-14T07:49:27.6264399Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4f5d6ef036f5032.xml (deflated 46%) 2025-03-14T07:49:27.6265879Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d5ccbe7f4758067.xml (deflated 46%) 2025-03-14T07:49:27.6267366Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-668bdf156270b40a.xml (deflated 44%) 2025-03-14T07:49:27.6268863Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4367dccd8f2d6b16.xml (deflated 43%) 2025-03-14T07:49:27.6270746Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf917650754763bd.xml (deflated 43%) 2025-03-14T07:49:27.6272271Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b2c194189cc98e.xml (deflated 43%) 2025-03-14T07:49:27.6273902Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0cb459036203fb47.xml (deflated 43%) 2025-03-14T07:49:27.6275419Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3bc910a5e5375138.xml (deflated 38%) 2025-03-14T07:49:27.6276951Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e3a959d11db5bce.xml (deflated 38%) 2025-03-14T07:49:27.6278479Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f6a7896009866556.xml (deflated 38%) 2025-03-14T07:49:27.6280002Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2a1457cfec3582c.xml (deflated 37%) 2025-03-14T07:49:27.6281521Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1f26ff18c2c8539a.xml (deflated 38%) 2025-03-14T07:49:27.6283044Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-965c15570a2c624b.xml (deflated 44%) 2025-03-14T07:49:27.6284717Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-711661fb779ffe92.xml (deflated 38%) 2025-03-14T07:49:27.6286312Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d5e847b8e6a6165.xml (deflated 47%) 2025-03-14T07:49:27.6287803Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df65dbf05f6d3570.xml (deflated 37%) 2025-03-14T07:49:27.6289288Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-181e7da08c287961.xml (deflated 38%) 2025-03-14T07:49:27.6290778Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-82f0ed805d194d05.xml (deflated 37%) 2025-03-14T07:49:27.6292262Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-901ef66e6fd34418.xml (deflated 37%) 2025-03-14T07:49:27.6293744Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c063b7ff6ba92c22.xml (deflated 38%) 2025-03-14T07:49:27.6295321Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5fd7ed1ef174e4a0.xml (deflated 38%) 2025-03-14T07:49:27.6296820Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8bb6c7d34df62a3b.xml (deflated 47%) 2025-03-14T07:49:27.6298296Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e579d0a2cac1fb1.xml (deflated 38%) 2025-03-14T07:49:27.6299779Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cfcc7729e4b71b5c.xml (deflated 39%) 2025-03-14T07:49:27.6301264Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f23c789ab35c1139.xml (deflated 38%) 2025-03-14T07:49:27.6302754Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-daed25eb3b5751dc.xml (deflated 44%) 2025-03-14T07:49:27.6304277Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71460d875bc6e165.xml (deflated 38%) 2025-03-14T07:49:27.6305748Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24342a309b24322b.xml (deflated 37%) 2025-03-14T07:49:27.6307214Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a00e033401629732.xml (deflated 43%) 2025-03-14T07:49:27.6308687Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1436abf48693cd58.xml (deflated 44%) 2025-03-14T07:49:27.6310167Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebb4bf933265fbaf.xml (deflated 45%) 2025-03-14T07:49:27.6311656Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aab1f949bc0bb208.xml (deflated 45%) 2025-03-14T07:49:27.6313203Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a549233e08d2e01.xml (deflated 46%) 2025-03-14T07:49:27.6314899Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8a8dd1dad4ddc6c.xml (deflated 45%) 2025-03-14T07:49:27.6316434Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e393230fd897a71.xml (deflated 38%) 2025-03-14T07:49:27.6317954Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bcc25794e6eacc31.xml (deflated 37%) 2025-03-14T07:49:27.6319550Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-81df4b2059afd699.xml (deflated 37%) 2025-03-14T07:49:27.6321082Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a39897df9c6b56d6.xml (deflated 37%) 2025-03-14T07:49:27.6322604Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ab1b5921ad75fec.xml (deflated 37%) 2025-03-14T07:49:27.6324126Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4deb017428c74945.xml (deflated 43%) 2025-03-14T07:49:27.6325737Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dca98713713c2d9d.xml (deflated 44%) 2025-03-14T07:49:27.6327201Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3e1e0088c089044.xml (deflated 44%) 2025-03-14T07:49:27.6328672Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-168a1a8e8f06e0df.xml (deflated 43%) 2025-03-14T07:49:27.6330221Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ab971eeb9cae903.xml (deflated 43%) 2025-03-14T07:49:27.6331708Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49df5efdd70dbc7d.xml (deflated 43%) 2025-03-14T07:49:27.6333180Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-523a91d14a6a6687.xml (deflated 44%) 2025-03-14T07:49:27.6334653Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a0a3c61831b62a3.xml (deflated 41%) 2025-03-14T07:49:27.6336135Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86362bd694fab90f.xml (deflated 38%) 2025-03-14T07:49:27.6337601Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4448e457238ad890.xml (deflated 38%) 2025-03-14T07:49:27.6339063Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-355117833963f4e5.xml (deflated 38%) 2025-03-14T07:49:27.6340530Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-322a9574b242dfe3.xml (deflated 37%) 2025-03-14T07:49:27.6341993Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bb7db05720dd047.xml (deflated 37%) 2025-03-14T07:49:27.6343463Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6885993c228b89c4.xml (deflated 38%) 2025-03-14T07:49:27.6344940Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17d8fe17e8eab116.xml (deflated 38%) 2025-03-14T07:49:27.6346403Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b16a9864b174021.xml (deflated 37%) 2025-03-14T07:49:27.6347860Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7375ac4bb8992519.xml (deflated 43%) 2025-03-14T07:49:27.6349334Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d16abea7ebb5dcd6.xml (deflated 38%) 2025-03-14T07:49:27.6350807Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1fa3e49616a4c50d.xml (deflated 37%) 2025-03-14T07:49:27.6352323Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58465f0f7dce7835.xml (deflated 37%) 2025-03-14T07:49:27.6354031Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a66eeee37844966a.xml (deflated 43%) 2025-03-14T07:49:27.6355550Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d8d7ab2181a99b09.xml (deflated 37%) 2025-03-14T07:49:27.6357070Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6fbca7ae0d90018.xml (deflated 46%) 2025-03-14T07:49:27.6358591Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab6c27d208da5e7e.xml (deflated 45%) 2025-03-14T07:49:27.6360107Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-506c0ebb8314e286.xml (deflated 44%) 2025-03-14T07:49:27.6361618Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9467a0035e2d0325.xml (deflated 38%) 2025-03-14T07:49:27.6363198Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-734e99b9182ffffc.xml (deflated 38%) 2025-03-14T07:49:27.6364707Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7227850d95d7577c.xml (deflated 37%) 2025-03-14T07:49:27.6366285Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e62bf4c694a0c495.xml (deflated 44%) 2025-03-14T07:49:27.6367748Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c155c4010661454.xml (deflated 44%) 2025-03-14T07:49:27.6369209Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e694f4bd5867d55.xml (deflated 44%) 2025-03-14T07:49:27.6371021Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29ff21ca4eca33d9.xml (deflated 43%) 2025-03-14T07:49:27.6372541Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afda69ecdeefd9b6.xml (deflated 44%) 2025-03-14T07:49:27.6374065Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4052d90caf26d5a4.xml (deflated 44%) 2025-03-14T07:49:27.6375576Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1c762d4e9560a53.xml (deflated 48%) 2025-03-14T07:49:27.6377088Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-675a7d057fc9353f.xml (deflated 38%) 2025-03-14T07:49:27.6378606Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6370d1548c587b5.xml (deflated 43%) 2025-03-14T07:49:27.6380110Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4cba21d968f6dcf8.xml (deflated 43%) 2025-03-14T07:49:27.6381770Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3be9776f24f470bf.xml (deflated 38%) 2025-03-14T07:49:27.6383242Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7f4cbfb7e4e9ad7.xml (deflated 37%) 2025-03-14T07:49:27.6384720Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91a6f3e85c7b7b27.xml (deflated 38%) 2025-03-14T07:49:27.6386182Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30111a26784deb3d.xml (deflated 37%) 2025-03-14T07:49:27.6387755Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-733cc526ecd5ff8e.xml (deflated 38%) 2025-03-14T07:49:27.6389234Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-117fffde5eb2dae5.xml (deflated 43%) 2025-03-14T07:49:27.6390706Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6300de9cc256c997.xml (deflated 38%) 2025-03-14T07:49:27.6392183Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd52ebdf3b5cd482.xml (deflated 58%) 2025-03-14T07:49:27.6393893Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bcac9b46ae811827.xml (deflated 38%) 2025-03-14T07:49:27.6395414Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96e56cc438ec9b05.xml (deflated 38%) 2025-03-14T07:49:27.6396929Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-74fb0a6179407065.xml (deflated 38%) 2025-03-14T07:49:27.6398523Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3fa79a0c7b6e60d3.xml (deflated 37%) 2025-03-14T07:49:27.6400039Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d1e45233eb385c8.xml (deflated 44%) 2025-03-14T07:49:27.6401559Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70e2b6e5446dbc68.xml (deflated 38%) 2025-03-14T07:49:27.6403072Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8019b546027b7e33.xml (deflated 43%) 2025-03-14T07:49:27.6404589Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fba033fd381d0d80.xml (deflated 38%) 2025-03-14T07:49:27.6406192Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f476aa965b61dda.xml (deflated 39%) 2025-03-14T07:49:27.6407662Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a3dca9125cbe4a9.xml (deflated 58%) 2025-03-14T07:49:27.6409142Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f73ae78b2e2e23b8.xml (deflated 38%) 2025-03-14T07:49:27.6410617Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76f77e4e901d9fa7.xml (deflated 44%) 2025-03-14T07:49:27.6412085Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-855ffdd4df121b25.xml (deflated 43%) 2025-03-14T07:49:27.6413550Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f4e7d8186057be2.xml (deflated 44%) 2025-03-14T07:49:27.6415002Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c816087081932bf3.xml (deflated 49%) 2025-03-14T07:49:27.6416469Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3cfdcd9cd5c61a41.xml (deflated 37%) 2025-03-14T07:49:27.6417943Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc83fa7f2458c636.xml (deflated 37%) 2025-03-14T07:49:27.6419404Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0901d97e744802ce.xml (deflated 37%) 2025-03-14T07:49:27.6420928Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-31e22f669f35996a.xml (deflated 37%) 2025-03-14T07:49:27.6422391Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec833335f1a4bd63.xml (deflated 38%) 2025-03-14T07:49:27.6423852Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe49a01712fc1e50.xml (deflated 38%) 2025-03-14T07:49:27.6425322Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-decab49461e8ee9d.xml (deflated 37%) 2025-03-14T07:49:27.6426795Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de84c2608ac70651.xml (deflated 38%) 2025-03-14T07:49:27.6428263Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a75ee92f8cb30eff.xml (deflated 37%) 2025-03-14T07:49:27.6429740Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8672cac4f07be9dc.xml (deflated 37%) 2025-03-14T07:49:27.6431259Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a4faee6ada69046.xml (deflated 37%) 2025-03-14T07:49:27.6432734Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6580c4715ca5734.xml (deflated 43%) 2025-03-14T07:49:27.6434500Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef3dabbad871c0c0.xml (deflated 43%) 2025-03-14T07:49:27.6436132Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90f811c080601b28.xml (deflated 43%) 2025-03-14T07:49:27.6437670Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e21c81a7fbe40b11.xml (deflated 43%) 2025-03-14T07:49:27.6439210Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c207762d53ed56d9.xml (deflated 45%) 2025-03-14T07:49:27.6440746Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ef6c0b3f7801361.xml (deflated 38%) 2025-03-14T07:49:27.6442281Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0fc7607518065f0a.xml (deflated 39%) 2025-03-14T07:49:27.6443828Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a08f86f31f705dba.xml (deflated 38%) 2025-03-14T07:49:27.6445369Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c12677cc64f0ba9.xml (deflated 37%) 2025-03-14T07:49:27.6446998Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80f07eb43ea4258c.xml (deflated 38%) 2025-03-14T07:49:27.6448496Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ebdcb953350fcf9.xml (deflated 46%) 2025-03-14T07:49:27.6450010Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cdbbcb962bf6bd33.xml (deflated 46%) 2025-03-14T07:49:27.6451519Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5340ca6e6ba6ddcc.xml (deflated 44%) 2025-03-14T07:49:27.6453027Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-512afd938b441dad.xml (deflated 43%) 2025-03-14T07:49:27.6454531Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b65f4a66c6be9094.xml (deflated 43%) 2025-03-14T07:49:27.6456085Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92240d5f37a97bc3.xml (deflated 43%) 2025-03-14T07:49:27.6457589Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6343e93f2206c293.xml (deflated 43%) 2025-03-14T07:49:27.6459084Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57d296a139b69473.xml (deflated 37%) 2025-03-14T07:49:27.6460578Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed7dc7140b125856.xml (deflated 38%) 2025-03-14T07:49:27.6462079Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7b2daf62614c2ee.xml (deflated 38%) 2025-03-14T07:49:27.6463577Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5935bd375b315e9c.xml (deflated 37%) 2025-03-14T07:49:27.6465080Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c780d6c4298742e2.xml (deflated 38%) 2025-03-14T07:49:27.6466852Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7591a6206102c80a.xml (deflated 45%) 2025-03-14T07:49:27.6468366Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37bf0e28683e7c21.xml (deflated 38%) 2025-03-14T07:49:27.6469976Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-72d9307966471902.xml (deflated 47%) 2025-03-14T07:49:27.6471566Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1854c27b2f313b4f.xml (deflated 38%) 2025-03-14T07:49:27.6473127Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1f1b3f8c47b007c.xml (deflated 38%) 2025-03-14T07:49:27.6474888Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8aa6f0ad236f6577.xml (deflated 37%) 2025-03-14T07:49:27.6476425Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bfc53297466e6965.xml (deflated 37%) 2025-03-14T07:49:27.6477966Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fdece5bbd203c98a.xml (deflated 37%) 2025-03-14T07:49:27.6479507Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7e043acd3c311a7.xml (deflated 38%) 2025-03-14T07:49:27.6481051Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-87c068c7c2866cc4.xml (deflated 47%) 2025-03-14T07:49:27.6482591Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fcf9720353929d93.xml (deflated 38%) 2025-03-14T07:49:27.6484141Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f07572e89b011e4.xml (deflated 38%) 2025-03-14T07:49:27.6485763Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e0a69f26dc84eda2.xml (deflated 38%) 2025-03-14T07:49:27.6487263Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10c78ce9a2ac12a6.xml (deflated 44%) 2025-03-14T07:49:27.6488764Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63fde346d053e5ec.xml (deflated 38%) 2025-03-14T07:49:27.6490367Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bbb240327266369c.xml (deflated 37%) 2025-03-14T07:49:27.6491865Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21e32545ba7af623.xml (deflated 43%) 2025-03-14T07:49:27.6493363Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ba52648b3732823.xml (deflated 43%) 2025-03-14T07:49:27.6494858Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cdfbb84c853f787e.xml (deflated 45%) 2025-03-14T07:49:27.6496353Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-391d9e65e7c55d26.xml (deflated 45%) 2025-03-14T07:49:27.6497842Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3c86e95ac646e13.xml (deflated 46%) 2025-03-14T07:49:27.6499344Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66639428dafe2f3d.xml (deflated 45%) 2025-03-14T07:49:27.6500908Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-46c3688106f1cd85.xml (deflated 37%) 2025-03-14T07:49:27.6502403Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62839605b05ddb34.xml (deflated 37%) 2025-03-14T07:49:27.6503896Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-31ac021e81410242.xml (deflated 37%) 2025-03-14T07:49:27.6505397Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28b701200134e348.xml (deflated 37%) 2025-03-14T07:49:27.6506891Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9989e339e22d08e.xml (deflated 37%) 2025-03-14T07:49:27.6508392Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c53de3ec61eb12e4.xml (deflated 43%) 2025-03-14T07:49:27.6509886Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9937d01f0bf265ea.xml (deflated 44%) 2025-03-14T07:49:27.6511385Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5c83597f90b38f1.xml (deflated 44%) 2025-03-14T07:49:27.6512874Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be47e11c8fe14368.xml (deflated 43%) 2025-03-14T07:49:27.6514700Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0a2aeb28b885441.xml (deflated 43%) 2025-03-14T07:49:27.6516250Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25cc8b3b3db37739.xml (deflated 43%) 2025-03-14T07:49:27.6517803Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-81d7e59f8dfc3efe.xml (deflated 44%) 2025-03-14T07:49:27.6519353Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4f43e353c6d4d6a.xml (deflated 41%) 2025-03-14T07:49:27.6520903Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-264da0b8d8cf211a.xml (deflated 37%) 2025-03-14T07:49:27.6522442Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c1a62877462ae9b.xml (deflated 38%) 2025-03-14T07:49:27.6523974Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2067395547a9a69c.xml (deflated 38%) 2025-03-14T07:49:27.6525684Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9fb1d2fbda96187.xml (deflated 37%) 2025-03-14T07:49:27.6527203Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4e165eca025ed96.xml (deflated 37%) 2025-03-14T07:49:27.6528703Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e2c1fda1a6a61abf.xml (deflated 38%) 2025-03-14T07:49:27.6530204Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3cac404ab73ae4f.xml (deflated 38%) 2025-03-14T07:49:27.6531708Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e08e814add96038.xml (deflated 37%) 2025-03-14T07:49:27.6533209Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a499a71fa8898ead.xml (deflated 44%) 2025-03-14T07:49:27.6534709Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b75fed8285a430b7.xml (deflated 38%) 2025-03-14T07:49:27.6536264Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca0329b4a11cc853.xml (deflated 38%) 2025-03-14T07:49:27.6537761Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-974b07561dbbc86a.xml (deflated 37%) 2025-03-14T07:49:27.6539254Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ca7a3996f2909f1.xml (deflated 43%) 2025-03-14T07:49:27.6540747Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37217257bbf667ec.xml (deflated 37%) 2025-03-14T07:49:27.6542251Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9aa09fa58abfb034.xml (deflated 45%) 2025-03-14T07:49:27.6543754Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef5799b4ffd100bb.xml (deflated 45%) 2025-03-14T07:49:27.6545252Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20be77bb7c3bac15.xml (deflated 44%) 2025-03-14T07:49:27.6546750Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4257f6053e979e4.xml (deflated 38%) 2025-03-14T07:49:27.6548248Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07e79ce5f981b7d8.xml (deflated 38%) 2025-03-14T07:49:27.6549747Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a669013e0a4c6af.xml (deflated 37%) 2025-03-14T07:49:27.6551250Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f4f748646e3f7f4.xml (deflated 44%) 2025-03-14T07:49:27.6552748Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd4adaee5e1698ae.xml (deflated 44%) 2025-03-14T07:49:27.6554512Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bea076f5e8acf393.xml (deflated 44%) 2025-03-14T07:49:27.6556057Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d7fbfb07def5d0e.xml (deflated 43%) 2025-03-14T07:49:27.6557608Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cfa6ed7173f6e066.xml (deflated 44%) 2025-03-14T07:49:27.6559215Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64b1e6de276802cb.xml (deflated 44%) 2025-03-14T07:49:27.6560760Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ea17bb7705c17bd.xml (deflated 48%) 2025-03-14T07:49:27.6562303Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c04e3d77d14d1f2.xml (deflated 38%) 2025-03-14T07:49:27.6563846Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aedb8235eaacdbc1.xml (deflated 43%) 2025-03-14T07:49:27.6565493Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a66cabd78c1e6a9a.xml (deflated 43%) 2025-03-14T07:49:27.6567267Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d49e6e2ac703812.xml (deflated 38%) 2025-03-14T07:49:27.6568763Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f26ad570fc48b0b.xml (deflated 37%) 2025-03-14T07:49:27.6570477Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-889cd5ea181f4faf.xml (deflated 37%) 2025-03-14T07:49:27.6571966Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02d101226c8c643c.xml (deflated 37%) 2025-03-14T07:49:27.6573449Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fec7689ab500623c.xml (deflated 37%) 2025-03-14T07:49:27.6574942Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8787fd10caf02ee.xml (deflated 43%) 2025-03-14T07:49:27.6576436Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30ff21f548e354aa.xml (deflated 38%) 2025-03-14T07:49:27.6577931Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2aa6a52ad8de6e90.xml (deflated 58%) 2025-03-14T07:49:27.6579426Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b0bb4ac4e80749a.xml (deflated 38%) 2025-03-14T07:49:27.6580913Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e0642891bae1aa49.xml (deflated 37%) 2025-03-14T07:49:27.6582498Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d07088718f124a71.xml (deflated 37%) 2025-03-14T07:49:27.6583942Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f35767da5135814.xml (deflated 37%) 2025-03-14T07:49:27.6585393Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3dbdac040cc9c337.xml (deflated 44%) 2025-03-14T07:49:27.6586854Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f08b1f616e2195e.xml (deflated 38%) 2025-03-14T07:49:27.6588298Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4a1270505fb35f5.xml (deflated 43%) 2025-03-14T07:49:27.6589737Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6066508a392bdbb0.xml (deflated 38%) 2025-03-14T07:49:27.6591184Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d452a4e612ba9c4c.xml (deflated 39%) 2025-03-14T07:49:27.6592723Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96c0a7559f821adc.xml (deflated 58%) 2025-03-14T07:49:27.6594452Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c87fa96616dc6e2b.xml (deflated 38%) 2025-03-14T07:49:27.6595996Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-964a3c6d9979bdce.xml (deflated 44%) 2025-03-14T07:49:27.6597538Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f8adffec9d713d7.xml (deflated 43%) 2025-03-14T07:49:27.6599075Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89ecf7e0d343b132.xml (deflated 44%) 2025-03-14T07:49:27.6600603Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b3142ae211c49b4.xml (deflated 49%) 2025-03-14T07:49:27.6602140Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e783756c0906b330.xml (deflated 37%) 2025-03-14T07:49:27.6603720Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8207a641ccc7813e.xml (deflated 37%) 2025-03-14T07:49:27.6605354Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f41b7fa72b99a113.xml (deflated 37%) 2025-03-14T07:49:27.6606916Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd27eadb2798280f.xml (deflated 38%) 2025-03-14T07:49:27.6608355Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ca0004b5e673c6d.xml (deflated 38%) 2025-03-14T07:49:27.6609798Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ddbcaec96e5eab3.xml (deflated 38%) 2025-03-14T07:49:27.6611256Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cf480de81acae65.xml (deflated 38%) 2025-03-14T07:49:27.6612714Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70bc2683204984be.xml (deflated 38%) 2025-03-14T07:49:27.6614152Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a890420e86aa082b.xml (deflated 37%) 2025-03-14T07:49:27.6615591Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-845aaa59bc93f172.xml (deflated 37%) 2025-03-14T07:49:27.6617030Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-68c0759232ffd0ee.xml (deflated 37%) 2025-03-14T07:49:27.6618473Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-320bf0c4cc2a925f.xml (deflated 43%) 2025-03-14T07:49:27.6619912Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bedc0404986023b3.xml (deflated 43%) 2025-03-14T07:49:27.6621353Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ec0c5a7d4d56844.xml (deflated 43%) 2025-03-14T07:49:27.6622794Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b40bfc11ab4783e0.xml (deflated 43%) 2025-03-14T07:49:27.6624232Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05f2f4d8c8aa0969.xml (deflated 45%) 2025-03-14T07:49:27.6625671Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae8df42b36814130.xml (deflated 38%) 2025-03-14T07:49:27.6627160Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51a9ec917b876f36.xml (deflated 39%) 2025-03-14T07:49:27.6628592Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3cb58933a906bd3.xml (deflated 38%) 2025-03-14T07:49:27.6630023Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c58c7698b5be7e3e.xml (deflated 37%) 2025-03-14T07:49:27.6631451Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-574497f7b1b42a90.xml (deflated 38%) 2025-03-14T07:49:27.6632951Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20d4be0bc0ec69c6.xml (deflated 38%) 2025-03-14T07:49:27.6634668Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40f90a26c2aa1a25.xml (deflated 38%) 2025-03-14T07:49:27.6636190Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b063bc5ee705aca.xml (deflated 37%) 2025-03-14T07:49:27.6637778Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f54bab78fcdb9fcc.xml (deflated 38%) 2025-03-14T07:49:27.6639295Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e229640ea893223.xml (deflated 38%) 2025-03-14T07:49:27.6640808Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-620de6ecf89ae178.xml (deflated 49%) 2025-03-14T07:49:27.6642331Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-035b49d1085b614c.xml (deflated 38%) 2025-03-14T07:49:27.6643846Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17f04fa9232b8a4d.xml (deflated 37%) 2025-03-14T07:49:27.6645470Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a34f09244429577b.xml (deflated 45%) 2025-03-14T07:49:27.6646950Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e1411a067e4c9ea7.xml (deflated 45%) 2025-03-14T07:49:27.6648316Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9f718b375c07cff.xml (deflated 44%) 2025-03-14T07:49:27.6649665Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-974bb4475b6d705b.xml (deflated 38%) 2025-03-14T07:49:27.6651007Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54382facbb657651.xml (deflated 50%) 2025-03-14T07:49:27.6652345Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95a2108d6ba9ee14.xml (deflated 42%) 2025-03-14T07:49:27.6653696Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7e7828836919d1e.xml (deflated 47%) 2025-03-14T07:49:27.6655039Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecf4333f67a963dd.xml (deflated 47%) 2025-03-14T07:49:27.6656389Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-35022fafd982af2e.xml (deflated 38%) 2025-03-14T07:49:27.6657746Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e9cdce3f789ccff.xml (deflated 37%) 2025-03-14T07:49:27.6659158Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e013c86f03c37867.xml (deflated 37%) 2025-03-14T07:49:27.6660504Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-411b1ba840a25bf3.xml (deflated 38%) 2025-03-14T07:49:27.6661859Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4233bb95f446a3b.xml (deflated 38%) 2025-03-14T07:49:27.6663212Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59eeec5db39e4392.xml (deflated 47%) 2025-03-14T07:49:27.6664565Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca81afaba9ec3842.xml (deflated 37%) 2025-03-14T07:49:27.6665916Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5fcd40b772ab8c5f.xml (deflated 38%) 2025-03-14T07:49:27.6667268Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d0524f7a75b079b.xml (deflated 37%) 2025-03-14T07:49:27.6668671Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-468a5d476a8df4b5.xml (deflated 38%) 2025-03-14T07:49:27.6670350Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8349a9ec76ff1981.xml (deflated 38%) 2025-03-14T07:49:27.6671824Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e7e57ac8c93fb2b.xml (deflated 37%) 2025-03-14T07:49:27.6673546Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-082316ded1f40346.xml (deflated 37%) 2025-03-14T07:49:27.6675059Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7f2879b116625e2.xml (deflated 38%) 2025-03-14T07:49:27.6676584Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dee33cb23d5a7ce9.xml (deflated 38%) 2025-03-14T07:49:27.6678112Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89eac6cd282ec1ae.xml (deflated 38%) 2025-03-14T07:49:27.6679635Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1eb2efb9abce76c5.xml (deflated 38%) 2025-03-14T07:49:27.6681158Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c807d645685a77fe.xml (deflated 37%) 2025-03-14T07:49:27.6682680Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b51d9f975c1f1bed.xml (deflated 38%) 2025-03-14T07:49:27.6684198Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44f378b722e18492.xml (deflated 37%) 2025-03-14T07:49:27.6685888Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-356f6645f17716da.xml (deflated 37%) 2025-03-14T07:49:27.6687313Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eeaf4418e7bb59fa.xml (deflated 37%) 2025-03-14T07:49:27.6688748Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c049cc5b29bd0e7.xml (deflated 43%) 2025-03-14T07:49:27.6690177Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-050b28001d041758.xml (deflated 38%) 2025-03-14T07:49:27.6691595Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e399bdb62551891.xml (deflated 37%) 2025-03-14T07:49:27.6693126Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc027e1bde125695.xml (deflated 37%) 2025-03-14T07:49:27.6694559Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37bd16faf1d9851e.xml (deflated 49%) 2025-03-14T07:49:27.6695994Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-edc1c540de345b68.xml (deflated 49%) 2025-03-14T07:49:27.6697423Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c13fa665bd94397.xml (deflated 43%) 2025-03-14T07:49:27.6698855Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66be85e7d7b7a7fa.xml (deflated 44%) 2025-03-14T07:49:27.6700294Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b97733816239e99.xml (deflated 37%) 2025-03-14T07:49:27.6701719Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b5004d0b29cac63.xml (deflated 38%) 2025-03-14T07:49:27.6703217Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-246f9280382027ed.xml (deflated 38%) 2025-03-14T07:49:27.6704641Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-197344da12439b86.xml (deflated 38%) 2025-03-14T07:49:27.6706139Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-391ea840cf98c98d.xml (deflated 37%) 2025-03-14T07:49:27.6707490Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-111007b91e7951ae.xml (deflated 37%) 2025-03-14T07:49:27.6708853Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf013b5f0a871383.xml (deflated 38%) 2025-03-14T07:49:27.6710199Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca797eb441e33028.xml (deflated 38%) 2025-03-14T07:49:27.6711555Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24fbeb2d658ddaab.xml (deflated 38%) 2025-03-14T07:49:27.6712958Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd48211d71cb8af2.xml (deflated 38%) 2025-03-14T07:49:27.6714648Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e58fe9ed9b0da40.xml (deflated 45%) 2025-03-14T07:49:27.6716169Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2189e8b5546edbb6.xml (deflated 45%) 2025-03-14T07:49:27.6717680Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-01fe2bc1448f101f.xml (deflated 37%) 2025-03-14T07:49:27.6719194Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bac628403905981.xml (deflated 49%) 2025-03-14T07:49:27.6720709Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a1e2e966cb798cd.xml (deflated 42%) 2025-03-14T07:49:27.6722219Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b950f81c0087dda4.xml (deflated 38%) 2025-03-14T07:49:27.6723739Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5063232aee81f6ae.xml (deflated 38%) 2025-03-14T07:49:27.6725398Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49e02f8473929a4c.xml (deflated 38%) 2025-03-14T07:49:27.6726881Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be7a0b73d3e98157.xml (deflated 38%) 2025-03-14T07:49:27.6728226Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32e94ad918f4c01c.xml (deflated 43%) 2025-03-14T07:49:27.6729564Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c8cc337e0e10467.xml (deflated 45%) 2025-03-14T07:49:27.6730904Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2da05fdc93d76a1.xml (deflated 38%) 2025-03-14T07:49:27.6732250Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2c4a69c5785ecbb.xml (deflated 48%) 2025-03-14T07:49:27.6733599Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ee54253e8169f34.xml (deflated 38%) 2025-03-14T07:49:27.6734994Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b899b951d1819c2.xml (deflated 43%) 2025-03-14T07:49:27.6736336Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-530a528019b13429.xml (deflated 44%) 2025-03-14T07:49:27.6737675Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-77b971d1d44b9c95.xml (deflated 44%) 2025-03-14T07:49:27.6739011Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-23d21e59d0293140.xml (deflated 44%) 2025-03-14T07:49:27.6740359Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac25f7aa51eb37b3.xml (deflated 38%) 2025-03-14T07:49:27.6741706Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec656157d04ebf6b.xml (deflated 37%) 2025-03-14T07:49:27.6743060Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-68e9794e2ff54e41.xml (deflated 38%) 2025-03-14T07:49:27.6744402Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2315cef1e1e7d13.xml (deflated 48%) 2025-03-14T07:49:27.6745742Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21b56e64abe541c0.xml (deflated 37%) 2025-03-14T07:49:27.6747087Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4643b8847c47e74b.xml (deflated 38%) 2025-03-14T07:49:27.6748438Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d14fb746ef77e8d.xml (deflated 37%) 2025-03-14T07:49:27.6749788Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb004bb9eea34704.xml (deflated 38%) 2025-03-14T07:49:27.6751146Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bda0e3228fb710b3.xml (deflated 37%) 2025-03-14T07:49:27.6752491Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13f9a500653765f6.xml (deflated 38%) 2025-03-14T07:49:27.6754118Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb0ebdd3a042486e.xml (deflated 43%) 2025-03-14T07:49:27.6755631Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7ef061fdeddbfc0.xml (deflated 38%) 2025-03-14T07:49:27.6757208Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5fe803d5e5797d8.xml (deflated 38%) 2025-03-14T07:49:27.6758723Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ae053a504d09219.xml (deflated 37%) 2025-03-14T07:49:27.6760233Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62bc3e35a946b50c.xml (deflated 38%) 2025-03-14T07:49:27.6761740Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30dac48853faf205.xml (deflated 38%) 2025-03-14T07:49:27.6763263Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca81b4dafd2fbedb.xml (deflated 45%) 2025-03-14T07:49:27.6764791Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff039ce3b9cd8163.xml (deflated 38%) 2025-03-14T07:49:27.6766447Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c20311a3092c810.xml (deflated 41%) 2025-03-14T07:49:27.6767838Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8e39bf8d527f182.xml (deflated 39%) 2025-03-14T07:49:27.6769191Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea558173f469b83e.xml (deflated 58%) 2025-03-14T07:49:27.6771132Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cfc75b69ff0b7d6e.xml (deflated 38%) 2025-03-14T07:49:27.6772594Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40ce9a1618fe8705.xml (deflated 38%) 2025-03-14T07:49:27.6774060Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f4901b885236a5b6.xml (deflated 43%) 2025-03-14T07:49:27.6775533Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93c1fba3f33435f7.xml (deflated 44%) 2025-03-14T07:49:27.6776994Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3fd985b84a007cd.xml (deflated 48%) 2025-03-14T07:49:27.6778468Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fca73473cdf65b6.xml (deflated 38%) 2025-03-14T07:49:27.6779940Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e5b6b9b85f575b5.xml (deflated 37%) 2025-03-14T07:49:27.6781415Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6b40ec76fa2ea87.xml (deflated 37%) 2025-03-14T07:49:27.6783041Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da9254eb9e3c4ad0.xml (deflated 37%) 2025-03-14T07:49:27.6784385Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a5b2c3cbf0cce43.xml (deflated 38%) 2025-03-14T07:49:27.6785728Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-127ae7a3a1e94dc4.xml (deflated 43%) 2025-03-14T07:49:27.6787082Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b5171e4dad31b81.xml (deflated 38%) 2025-03-14T07:49:27.6788431Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-06a714cf5093e527.xml (deflated 45%) 2025-03-14T07:49:27.6789865Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70434775059c4afd.xml (deflated 37%) 2025-03-14T07:49:27.6791203Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e488c1690017392.xml (deflated 37%) 2025-03-14T07:49:27.6792557Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d7b95f18709a688.xml (deflated 37%) 2025-03-14T07:49:27.6794199Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37b2a3df80850dbc.xml (deflated 38%) 2025-03-14T07:49:27.6795720Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dceaecc010824ddd.xml (deflated 38%) 2025-03-14T07:49:27.6797241Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7f434bd0fcee0c5e.xml (deflated 38%) 2025-03-14T07:49:27.6798779Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f451baad25fd74ed.xml (deflated 49%) 2025-03-14T07:49:27.6800395Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5dae118f7d449c88.xml (deflated 37%) 2025-03-14T07:49:27.6801925Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29b40ce54d7e626b.xml (deflated 38%) 2025-03-14T07:49:27.6803463Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14eb70af4b25cabc.xml (deflated 40%) 2025-03-14T07:49:27.6805002Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f63f54ed7c93c90.xml (deflated 38%) 2025-03-14T07:49:27.6806545Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfc15f15a07a353f.xml (deflated 37%) 2025-03-14T07:49:27.6807916Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed1d3a1de9501e25.xml (deflated 38%) 2025-03-14T07:49:27.6809303Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6771ad74aadfe37a.xml (deflated 37%) 2025-03-14T07:49:27.6810681Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b8e2c4ebf9eae6b.xml (deflated 38%) 2025-03-14T07:49:27.6812057Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02e119bd4f80d682.xml (deflated 38%) 2025-03-14T07:49:27.6813429Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d89a60930cc8d9fd.xml (deflated 37%) 2025-03-14T07:49:27.6814802Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-03552fa4ac22c300.xml (deflated 37%) 2025-03-14T07:49:27.6816165Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12df970b8d3200d7.xml (deflated 49%) 2025-03-14T07:49:27.6817542Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4c427087b60da2c.xml (deflated 38%) 2025-03-14T07:49:27.6818914Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f56c1b754dd1653b.xml (deflated 37%) 2025-03-14T07:49:27.6820289Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e158ad4a3e4a9cd1.xml (deflated 45%) 2025-03-14T07:49:27.6821664Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bfa374016949a96d.xml (deflated 45%) 2025-03-14T07:49:27.6823092Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c661849efab64ea5.xml (deflated 44%) 2025-03-14T07:49:27.6824467Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90c05f74c7b85c30.xml (deflated 38%) 2025-03-14T07:49:27.6825839Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f428ac8146dada2e.xml (deflated 50%) 2025-03-14T07:49:27.6827219Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa93bc93c1ee289b.xml (deflated 42%) 2025-03-14T07:49:27.6828595Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96a269c21d3fca29.xml (deflated 47%) 2025-03-14T07:49:27.6829983Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf9d41bbe7777b34.xml (deflated 47%) 2025-03-14T07:49:27.6831359Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cb8b126413a294c.xml (deflated 37%) 2025-03-14T07:49:27.6832800Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1956ee9af72b2241.xml (deflated 37%) 2025-03-14T07:49:27.6834508Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1ddec24251f0077.xml (deflated 37%) 2025-03-14T07:49:27.6836043Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce2c3111d67c1caf.xml (deflated 37%) 2025-03-14T07:49:27.6837576Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13322f1c2687fd4d.xml (deflated 38%) 2025-03-14T07:49:27.6839119Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7bcaae3f05bc80de.xml (deflated 47%) 2025-03-14T07:49:27.6840670Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d4a9d5eae02bbde.xml (deflated 38%) 2025-03-14T07:49:27.6842202Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2ae3668aec5c905.xml (deflated 38%) 2025-03-14T07:49:27.6843739Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c90f3c0469595efe.xml (deflated 37%) 2025-03-14T07:49:27.6845275Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9176234a6274871.xml (deflated 38%) 2025-03-14T07:49:27.6846786Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d090c3be30dd995d.xml (deflated 38%) 2025-03-14T07:49:27.6848162Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ce00e5ddc94a072.xml (deflated 37%) 2025-03-14T07:49:27.6849527Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e1ed7a5e5ac2539.xml (deflated 37%) 2025-03-14T07:49:27.6850888Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-84b8f030ad6f3ea3.xml (deflated 38%) 2025-03-14T07:49:27.6852245Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cfc0a0a886a95898.xml (deflated 38%) 2025-03-14T07:49:27.6853608Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bd9ad57412646e6.xml (deflated 38%) 2025-03-14T07:49:27.6855039Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ac5b19d16289236.xml (deflated 38%) 2025-03-14T07:49:27.6856403Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a9fb4ad4b9791fd.xml (deflated 38%) 2025-03-14T07:49:27.6857772Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e178620d30e9a733.xml (deflated 38%) 2025-03-14T07:49:27.6859129Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-68392f1fce7dc052.xml (deflated 37%) 2025-03-14T07:49:27.6860488Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-69ab78650f15d57a.xml (deflated 38%) 2025-03-14T07:49:27.6861857Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b079e1bfaf5a18a.xml (deflated 37%) 2025-03-14T07:49:27.6863226Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3714ba0b85b1a80e.xml (deflated 43%) 2025-03-14T07:49:27.6864638Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89e716f799f3fd1a.xml (deflated 38%) 2025-03-14T07:49:27.6865999Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92e0743ddd16361f.xml (deflated 37%) 2025-03-14T07:49:27.6867349Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62f2fa67cd565053.xml (deflated 38%) 2025-03-14T07:49:27.6868711Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-264231c27ba9e186.xml (deflated 49%) 2025-03-14T07:49:27.6870424Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98975209ca5a6de0.xml (deflated 49%) 2025-03-14T07:49:27.6871913Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb05a4cc7a762ce2.xml (deflated 43%) 2025-03-14T07:49:27.6873641Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f1faed43649007f.xml (deflated 44%) 2025-03-14T07:49:27.6875185Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce4d12196ee28085.xml (deflated 37%) 2025-03-14T07:49:27.6876725Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42eaa1b8e9b4010b.xml (deflated 37%) 2025-03-14T07:49:27.6878261Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1953f6668a10e0f.xml (deflated 38%) 2025-03-14T07:49:27.6879806Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-216c470a9d1ac80e.xml (deflated 38%) 2025-03-14T07:49:27.6881338Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1943814c37c610b.xml (deflated 37%) 2025-03-14T07:49:27.6882883Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a31be1dcad0cf2ff.xml (deflated 37%) 2025-03-14T07:49:27.6884431Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c43d82838fa8ccd.xml (deflated 38%) 2025-03-14T07:49:27.6886110Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0a737514ca5ed552.xml (deflated 38%) 2025-03-14T07:49:27.6887484Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12126b757be29a8d.xml (deflated 37%) 2025-03-14T07:49:27.6888945Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-448d2fb5d92bda42.xml (deflated 38%) 2025-03-14T07:49:27.6890313Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3700aef110dc47b7.xml (deflated 45%) 2025-03-14T07:49:27.6891684Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-884b7b17ca82cdcf.xml (deflated 45%) 2025-03-14T07:49:27.6893050Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c621bf9a5233ca81.xml (deflated 37%) 2025-03-14T07:49:27.6894420Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1b8d55cb0213bb8.xml (deflated 49%) 2025-03-14T07:49:27.6895789Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-426fa750154679a1.xml (deflated 42%) 2025-03-14T07:49:27.6897210Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39a64205e45b71e5.xml (deflated 37%) 2025-03-14T07:49:27.6898581Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1ccedfd57077290.xml (deflated 38%) 2025-03-14T07:49:27.6899949Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2093efe1e25c4b7c.xml (deflated 38%) 2025-03-14T07:49:27.6901316Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ff3d9d91bedcc8d.xml (deflated 38%) 2025-03-14T07:49:27.6902680Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29885def00bde801.xml (deflated 43%) 2025-03-14T07:49:27.6904050Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2df78800568bd12.xml (deflated 45%) 2025-03-14T07:49:27.6905411Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0aa3f0cf00868056.xml (deflated 37%) 2025-03-14T07:49:27.6906772Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f112848bc1764821.xml (deflated 49%) 2025-03-14T07:49:27.6908123Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a8708868c65e68c.xml (deflated 38%) 2025-03-14T07:49:27.6909480Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c79fe5145f57a1e8.xml (deflated 43%) 2025-03-14T07:49:27.6910850Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b00ba691dd0794f.xml (deflated 44%) 2025-03-14T07:49:27.6912208Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e37efa123e8983e1.xml (deflated 44%) 2025-03-14T07:49:27.6913821Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21c196ac585991d7.xml (deflated 44%) 2025-03-14T07:49:27.6915354Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-792eb0e13f3cdd92.xml (deflated 38%) 2025-03-14T07:49:27.6916881Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9463615cd8d3b529.xml (deflated 38%) 2025-03-14T07:49:27.6918420Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb1835e254c94aa4.xml (deflated 38%) 2025-03-14T07:49:27.6920101Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3feb96fad048164.xml (deflated 47%) 2025-03-14T07:49:27.6921641Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ad9c809ff25a208.xml (deflated 37%) 2025-03-14T07:49:27.6923171Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c8a433010e142af.xml (deflated 38%) 2025-03-14T07:49:27.6924707Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4516fd45605f8d0c.xml (deflated 37%) 2025-03-14T07:49:27.6926302Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ebf5b1626acf262.xml (deflated 38%) 2025-03-14T07:49:27.6927669Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ec8672b9d157feb.xml (deflated 37%) 2025-03-14T07:49:27.6929034Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60f0c41aecd36782.xml (deflated 38%) 2025-03-14T07:49:27.6930451Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98f4904bec0488a0.xml (deflated 43%) 2025-03-14T07:49:27.6931819Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea6ed2431424ea2d.xml (deflated 38%) 2025-03-14T07:49:27.6933183Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ada744bb2a7734f5.xml (deflated 38%) 2025-03-14T07:49:27.6934563Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3814fff9facf89c.xml (deflated 37%) 2025-03-14T07:49:27.6935931Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92e55201f6a15a07.xml (deflated 37%) 2025-03-14T07:49:27.6937303Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba9d2c794e46bd00.xml (deflated 37%) 2025-03-14T07:49:27.6938676Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f85401784eeb440.xml (deflated 45%) 2025-03-14T07:49:27.6940042Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c40526286defda9.xml (deflated 38%) 2025-03-14T07:49:27.6941416Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f295631b7d6a464.xml (deflated 42%) 2025-03-14T07:49:27.6942775Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6412d5cb5328861.xml (deflated 39%) 2025-03-14T07:49:27.6944138Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0646abbbe92bbbdd.xml (deflated 58%) 2025-03-14T07:49:27.6945510Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78b61c42fa7bd4f8.xml (deflated 38%) 2025-03-14T07:49:27.6946882Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6adcdde95550b202.xml (deflated 38%) 2025-03-14T07:49:27.6948252Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10e9a32eff123fc6.xml (deflated 43%) 2025-03-14T07:49:27.6949617Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-216f6592c1264902.xml (deflated 44%) 2025-03-14T07:49:27.6951037Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-163377dfee6cf0c2.xml (deflated 49%) 2025-03-14T07:49:27.6952403Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ccd9e70edc018cc.xml (deflated 37%) 2025-03-14T07:49:27.6954058Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d19f443a45fdafa.xml (deflated 37%) 2025-03-14T07:49:27.6955595Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-31bcbb5d28b339ad.xml (deflated 37%) 2025-03-14T07:49:27.6957138Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45dd7ddd4c313f86.xml (deflated 37%) 2025-03-14T07:49:27.6958678Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f679e5e6a3ad834f.xml (deflated 38%) 2025-03-14T07:49:27.6960220Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da46ce3daa683512.xml (deflated 43%) 2025-03-14T07:49:27.6961877Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e4bc5bbb6d7cd94.xml (deflated 38%) 2025-03-14T07:49:27.6963424Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3965f57839488561.xml (deflated 45%) 2025-03-14T07:49:27.6964959Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aeefaeff5dfff847.xml (deflated 37%) 2025-03-14T07:49:27.6966513Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4eb4d9f73ca5ebfd.xml (deflated 37%) 2025-03-14T07:49:27.6967884Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1b57c27ddd970dc.xml (deflated 37%) 2025-03-14T07:49:27.6969365Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fde935bbbec0d3f6.xml (deflated 38%) 2025-03-14T07:49:27.6971014Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-629785683dee5b78.xml (deflated 38%) 2025-03-14T07:49:27.6972505Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e31e9fdfac2e8d2.xml (deflated 38%) 2025-03-14T07:49:27.6973994Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e7d665094abd984.xml (deflated 49%) 2025-03-14T07:49:27.6975475Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-713a8f6e89197fd6.xml (deflated 37%) 2025-03-14T07:49:27.6976965Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5fad4d539a3adf7.xml (deflated 38%) 2025-03-14T07:49:27.6978408Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-3de5876cc50b0495.xml (deflated 85%) 2025-03-14T07:49:27.6979935Z adding: test/test-reports/python-pytest/distributed.fsdp.test_distributed_checkpoint/distributed.fsdp.test_distributed_checkpoint-33022ddfa65b5aec.xml (deflated 76%) 2025-03-14T07:49:27.6981553Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_misc/distributed.fsdp.test_fsdp_misc-64b0b49c22632df0.xml (deflated 79%) 2025-03-14T07:49:27.6982905Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ea2b6d2874dfc3d1.xml (deflated 35%) 2025-03-14T07:49:27.6984052Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-88358de80b3a2199.xml (deflated 44%) 2025-03-14T07:49:27.6985290Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f0547ec8255dc007.xml (deflated 35%) 2025-03-14T07:49:27.6986447Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c558c9c09add39c8.xml (deflated 35%) 2025-03-14T07:49:27.6987601Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-196318b82a0086e9.xml (deflated 35%) 2025-03-14T07:49:27.6988756Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c9b741f8894179a1.xml (deflated 34%) 2025-03-14T07:49:27.6989914Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-be0dbec76db63c54.xml (deflated 35%) 2025-03-14T07:49:27.6991075Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-febf51b46c74eee3.xml (deflated 35%) 2025-03-14T07:49:27.6992230Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-df8780571585358c.xml (deflated 36%) 2025-03-14T07:49:27.6993621Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-dc40b9803a804773.xml (deflated 35%) 2025-03-14T07:49:27.6994992Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3b728215b3712625.xml (deflated 36%) 2025-03-14T07:49:27.6996293Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e58b2554cb0d4a26.xml (deflated 49%) 2025-03-14T07:49:27.6997593Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fbbad1618e02107b.xml (deflated 49%) 2025-03-14T07:49:27.6998895Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5844a7a3ebd9d801.xml (deflated 34%) 2025-03-14T07:49:27.7000196Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-69bfac4e57705a1c.xml (deflated 34%) 2025-03-14T07:49:27.7001506Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-89b4656dadcf0dff.xml (deflated 37%) 2025-03-14T07:49:27.7002803Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-aa175127635a0ba3.xml (deflated 35%) 2025-03-14T07:49:27.7004100Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-613acc3ef5621541.xml (deflated 37%) 2025-03-14T07:49:27.7005491Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-36236b9b565d6673.xml (deflated 36%) 2025-03-14T07:49:27.7006772Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5062ab29f5d1ade6.xml (deflated 36%) 2025-03-14T07:49:27.7007925Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5ee60aedc46e5ab7.xml (deflated 36%) 2025-03-14T07:49:27.7009079Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0bf553b637599169.xml (deflated 37%) 2025-03-14T07:49:27.7010227Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a9212b6de3556a29.xml (deflated 36%) 2025-03-14T07:49:27.7011384Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1acd9f196b55ceaa.xml (deflated 36%) 2025-03-14T07:49:27.7012534Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-490039b184fee05c.xml (deflated 37%) 2025-03-14T07:49:27.7013687Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-772be33e19bcb98f.xml (deflated 36%) 2025-03-14T07:49:27.7014837Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f22a1771a18d64b7.xml (deflated 36%) 2025-03-14T07:49:27.7016037Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-868117e1cb987346.xml (deflated 36%) 2025-03-14T07:49:27.7017185Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-777ffaf61b14a61a.xml (deflated 36%) 2025-03-14T07:49:27.7018337Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-29e05df972746a74.xml (deflated 37%) 2025-03-14T07:49:27.7019486Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-40a18d4cdcc460af.xml (deflated 36%) 2025-03-14T07:49:27.7020646Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6ae13ca14d518777.xml (deflated 36%) 2025-03-14T07:49:27.7021803Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-70330df9ca5945dd.xml (deflated 36%) 2025-03-14T07:49:27.7022958Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-619082a572eea918.xml (deflated 36%) 2025-03-14T07:49:27.7024110Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1c4efd1495bdffed.xml (deflated 44%) 2025-03-14T07:49:27.7025315Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1b2e1fe085be44ff.xml (deflated 37%) 2025-03-14T07:49:27.7026468Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0e70355cf61791d7.xml (deflated 37%) 2025-03-14T07:49:27.7027624Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a9323e215acac4be.xml (deflated 37%) 2025-03-14T07:49:27.7028779Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bbb2a76ad0d27954.xml (deflated 37%) 2025-03-14T07:49:27.7029931Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a7cea4f7820882dc.xml (deflated 36%) 2025-03-14T07:49:27.7031086Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f6c440831372f379.xml (deflated 36%) 2025-03-14T07:49:27.7032234Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fc9f70a8f625d9fe.xml (deflated 35%) 2025-03-14T07:49:27.7033653Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c0cc6d0e29cf18d9.xml (deflated 35%) 2025-03-14T07:49:27.7034950Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-892649719091eb5f.xml (deflated 36%) 2025-03-14T07:49:27.7036242Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3692a385b5779e6d.xml (deflated 35%) 2025-03-14T07:49:27.7037540Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2e2dadab3762e36d.xml (deflated 36%) 2025-03-14T07:49:27.7038847Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ad46a81c7c29e8d5.xml (deflated 36%) 2025-03-14T07:49:27.7040153Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1ce87576c32be57b.xml (deflated 35%) 2025-03-14T07:49:27.7041461Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-75638b1950691f06.xml (deflated 36%) 2025-03-14T07:49:27.7042760Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-dc937f469d415ae6.xml (deflated 44%) 2025-03-14T07:49:27.7044064Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-989970013965e2a4.xml (deflated 36%) 2025-03-14T07:49:27.7045474Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d42d4e53c963471c.xml (deflated 37%) 2025-03-14T07:49:27.7046775Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-135a5e7d31b36004.xml (deflated 37%) 2025-03-14T07:49:27.7048001Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-dfe596dc5bbf21c4.xml (deflated 36%) 2025-03-14T07:49:27.7049176Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0dc7e2a35da33acd.xml (deflated 37%) 2025-03-14T07:49:27.7050341Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1619218774e16619.xml (deflated 36%) 2025-03-14T07:49:27.7051501Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-87287fba26c85922.xml (deflated 35%) 2025-03-14T07:49:27.7052659Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5a2d7d6454b8c9a0.xml (deflated 36%) 2025-03-14T07:49:27.7053822Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-24f73bd50b93b0ee.xml (deflated 33%) 2025-03-14T07:49:27.7055000Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bf071594b18787d9.xml (deflated 34%) 2025-03-14T07:49:27.7056166Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-cbc7a81acbfeff97.xml (deflated 34%) 2025-03-14T07:49:27.7057384Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-108e000e7d418a03.xml (deflated 35%) 2025-03-14T07:49:27.7058545Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d923065291cb5d1.xml (deflated 35%) 2025-03-14T07:49:27.7059703Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-11204fe21890fbfc.xml (deflated 36%) 2025-03-14T07:49:27.7060867Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c8e603e64777cdac.xml (deflated 35%) 2025-03-14T07:49:27.7062026Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-859272f85b3c4776.xml (deflated 36%) 2025-03-14T07:49:27.7063191Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fba93b66c6e5b223.xml (deflated 35%) 2025-03-14T07:49:27.7064358Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-cfdd5b5fea68747b.xml (deflated 36%) 2025-03-14T07:49:27.7065521Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8f4f5712ce4f925e.xml (deflated 35%) 2025-03-14T07:49:27.7066676Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-db10722497fcb697.xml (deflated 35%) 2025-03-14T07:49:27.7067835Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a24d196f82a6ee8d.xml (deflated 36%) 2025-03-14T07:49:27.7068990Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-580b334cf2952205.xml (deflated 35%) 2025-03-14T07:49:27.7070490Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4920f1ce25c66875.xml (deflated 35%) 2025-03-14T07:49:27.7071756Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9da3e1ab67f4386a.xml (deflated 35%) 2025-03-14T07:49:27.7073091Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-770e21f3af277c59.xml (deflated 35%) 2025-03-14T07:49:27.7074557Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-38a964f4d77332ea.xml (deflated 35%) 2025-03-14T07:49:27.7075861Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f016e1706ba9fbb5.xml (deflated 35%) 2025-03-14T07:49:27.7077164Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e339963213df52d5.xml (deflated 34%) 2025-03-14T07:49:27.7078587Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6491c5934cd9a20c.xml (deflated 35%) 2025-03-14T07:49:27.7079893Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b1ca36e8bb2c5a7f.xml (deflated 35%) 2025-03-14T07:49:27.7081205Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b41ce59faf465fd8.xml (deflated 36%) 2025-03-14T07:49:27.7082519Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fc8d6ab9ef97b0cf.xml (deflated 34%) 2025-03-14T07:49:27.7083838Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b87af55decba8672.xml (deflated 42%) 2025-03-14T07:49:27.7085479Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-0be87f2ada64f4e1.xml (deflated 93%) 2025-03-14T07:49:27.7086778Z adding: test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-017bf75826cdd769.xml (deflated 87%) 2025-03-14T07:49:27.7087958Z adding: test/test-reports/python-pytest/distributed.test_c10d_logger/distributed.test_c10d_logger-2066e19c75dcc1a9.xml (deflated 48%) 2025-03-14T07:49:27.7089335Z adding: test/test-reports/python-pytest/distributed.test_c10d_object_collectives/distributed.test_c10d_object_collectives-aed0e84f9a97372c.xml (deflated 81%) 2025-03-14T07:49:27.7090627Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0af331d67fb32777.xml (deflated 39%) 2025-03-14T07:49:27.7091787Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8a2dde8b2d1cd9db.xml (deflated 38%) 2025-03-14T07:49:27.7092953Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5f9ef03cdc8580da.xml (deflated 37%) 2025-03-14T07:49:27.7094121Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-86fc524e6bc415e9.xml (deflated 37%) 2025-03-14T07:49:27.7095290Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-efda13b79e5ac2a2.xml (deflated 37%) 2025-03-14T07:49:27.7096463Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a467e2e4460447ac.xml (deflated 37%) 2025-03-14T07:49:27.7097626Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f3240bc3d049d7ad.xml (deflated 38%) 2025-03-14T07:49:27.7098785Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-93b55e267815f132.xml (deflated 38%) 2025-03-14T07:49:27.7099942Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-aa6273cc8274a488.xml (deflated 37%) 2025-03-14T07:49:27.7101098Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d96b5695bb6d04d9.xml (deflated 38%) 2025-03-14T07:49:27.7102259Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bc71e0c89e0c7cbe.xml (deflated 38%) 2025-03-14T07:49:27.7103419Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-78f2950318796f9f.xml (deflated 37%) 2025-03-14T07:49:27.7104577Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-359c92b3b7a343f1.xml (deflated 37%) 2025-03-14T07:49:27.7105740Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4f40ffc4c7695acb.xml (deflated 38%) 2025-03-14T07:49:27.7106896Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e1414d7ef3617793.xml (deflated 38%) 2025-03-14T07:49:27.7108048Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ea87d81037ef53d5.xml (deflated 38%) 2025-03-14T07:49:27.7109265Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-70e7f059635d1ba4.xml (deflated 38%) 2025-03-14T07:49:27.7110428Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3a93dda1fd251ddd.xml (deflated 38%) 2025-03-14T07:49:27.7111590Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-da4d478144794d11.xml (deflated 38%) 2025-03-14T07:49:27.7112743Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9f56e2421ae65710.xml (deflated 37%) 2025-03-14T07:49:27.7114220Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-880cd6dc016994d3.xml (deflated 38%) 2025-03-14T07:49:27.7115529Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-04a8f05b24498e4c.xml (deflated 38%) 2025-03-14T07:49:27.7116833Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c21e46ec3692a3e3.xml (deflated 38%) 2025-03-14T07:49:27.7118155Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bc84e9eac1415f9f.xml (deflated 37%) 2025-03-14T07:49:27.7119522Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-82814b2700467dfe.xml (deflated 37%) 2025-03-14T07:49:27.7120825Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f21685f10f5b3f8c.xml (deflated 38%) 2025-03-14T07:49:27.7122125Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5f884f255b649525.xml (deflated 38%) 2025-03-14T07:49:27.7123426Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-28fc3b849ca789bd.xml (deflated 38%) 2025-03-14T07:49:27.7124731Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7f5a1b1abb26cc12.xml (deflated 38%) 2025-03-14T07:49:27.7126204Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a7ecdbef2c6e5012.xml (deflated 38%) 2025-03-14T07:49:27.7127359Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-02c90c06e1727612.xml (deflated 39%) 2025-03-14T07:49:27.7128514Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5e92814857a04f9b.xml (deflated 38%) 2025-03-14T07:49:27.7129665Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-638a30c088b75d0c.xml (deflated 38%) 2025-03-14T07:49:27.7130821Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-123def4d778a7a1c.xml (deflated 38%) 2025-03-14T07:49:27.7131979Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-73f3b4374434e32e.xml (deflated 38%) 2025-03-14T07:49:27.7133140Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6616853816fe9c65.xml (deflated 37%) 2025-03-14T07:49:27.7134300Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b023618895cf52f4.xml (deflated 37%) 2025-03-14T07:49:27.7135458Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7037ff20922b813d.xml (deflated 38%) 2025-03-14T07:49:27.7136613Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7d29678c77599060.xml (deflated 38%) 2025-03-14T07:49:27.7137758Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-83978a6c647c9098.xml (deflated 38%) 2025-03-14T07:49:27.7138916Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a1474ce4efab1973.xml (deflated 38%) 2025-03-14T07:49:27.7140076Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-32212caeb9cfc74c.xml (deflated 38%) 2025-03-14T07:49:27.7141289Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5011adbe98b33174.xml (deflated 59%) 2025-03-14T07:49:27.7142451Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a072b44aaca148ff.xml (deflated 39%) 2025-03-14T07:49:27.7143615Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-41c882c362cab49a.xml (deflated 38%) 2025-03-14T07:49:27.7144774Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-da41d0cbb97a5be1.xml (deflated 38%) 2025-03-14T07:49:27.7146098Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f1285b1565862af4.xml (deflated 38%) 2025-03-14T07:49:27.7147321Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8501e931654b611f.xml (deflated 38%) 2025-03-14T07:49:27.7148550Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-613961f528cf8254.xml (deflated 38%) 2025-03-14T07:49:27.7149770Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cce0a76dc94e11b3.xml (deflated 38%) 2025-03-14T07:49:27.7151055Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2ee410f95462abf4.xml (deflated 59%) 2025-03-14T07:49:27.7152282Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a8ae7a95b1386919.xml (deflated 38%) 2025-03-14T07:49:27.7153777Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2bff7629f79bcff1.xml (deflated 36%) 2025-03-14T07:49:27.7155091Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4f9e24bb889188a4.xml (deflated 36%) 2025-03-14T07:49:27.7156405Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-577ef91b171a6ebb.xml (deflated 37%) 2025-03-14T07:49:27.7157715Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a7315fee47a60f95.xml (deflated 38%) 2025-03-14T07:49:27.7174102Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0b67b8c0a98876bf.xml (deflated 38%) 2025-03-14T07:49:27.7175388Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-85c89e7efba0a09b.xml (deflated 38%) 2025-03-14T07:49:27.7176647Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b5461ac941530ab3.xml (deflated 38%) 2025-03-14T07:49:27.7177906Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ed546762bc63ebfb.xml (deflated 38%) 2025-03-14T07:49:27.7179166Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5a171644b9ba14b2.xml (deflated 59%) 2025-03-14T07:49:27.7180429Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8ba0f2b7ffaef8ad.xml (deflated 36%) 2025-03-14T07:49:27.7181791Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ff79bde35d5733f8.xml (deflated 37%) 2025-03-14T07:49:27.7183017Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9b3e9e39124cf4cd.xml (deflated 37%) 2025-03-14T07:49:27.7184321Z adding: test/test-reports/python-pytest/distributed.tensor.test_attention/distributed.tensor.test_attention-687e075f249a8bec.xml (deflated 68%) 2025-03-14T07:49:27.7185843Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fsdp_ep/distributed.checkpoint.e2e.test_fsdp_ep-7516e2706a95a992.xml (deflated 50%) 2025-03-14T07:49:27.7187157Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fx/distributed.fsdp.test_fsdp_fx-85c357e361963198.xml (deflated 35%) 2025-03-14T07:49:27.7188880Z adding: test/test-reports/python-pytest/distributed.tensor.test_experimental_ops/distributed.tensor.test_experimental_ops-1e8bdb4d36711b9b.xml (deflated 57%) 2025-03-14T07:49:27.7190568Z adding: test/test-reports/python-pytest/distributed.tensor.experimental.test_register_sharding/distributed.tensor.experimental.test_register_sharding-ea3857ac054b5d3e.xml (deflated 50%) 2025-03-14T07:49:27.7192116Z adding: test/test-reports/python-pytest/distributed.test_backends/distributed.test_backends-210a98a4e8ea22e1.xml (deflated 49%) 2025-03-14T07:49:27.7193806Z adding: test/test-reports/python-pytest/distributed.checkpoint.fsdp.test_fsdp_dsd/distributed.checkpoint.fsdp.test_fsdp_dsd-e857e54525b02a5a.xml (deflated 72%) 2025-03-14T07:49:27.7195328Z adding: test/test-reports/python-pytest/distributed.tensor.test_init/distributed.tensor.test_init-9c1a8dec2094ac1a.xml (deflated 72%) 2025-03-14T07:49:27.7196871Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_optim_state/distributed.checkpoint.test_fsdp_optim_state-c8fea820bb02d9fb.xml (deflated 53%) 2025-03-14T07:49:27.7198555Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-ba8abc8e861b71f2.xml (deflated 48%) 2025-03-14T07:49:27.7200217Z adding: test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-c32844535d027465.xml (deflated 63%) 2025-03-14T07:49:27.7201723Z adding: test/test-reports/python-pytest/distributed.tensor.test_random_ops/distributed.tensor.test_random_ops-e6e101bb03d1e393.xml (deflated 77%) 2025-03-14T07:49:27.7203252Z adding: test/test-reports/python-pytest/distributed.pipelining.test_backward/distributed.pipelining.test_backward-c9d60cc6c057f920.xml (deflated 68%) 2025-03-14T07:49:27.7204780Z adding: test/test-reports/python-pytest/distributed.pipelining.test_pipe/distributed.pipelining.test_pipe-165f5e7bcd94aed7.xml (deflated 81%) 2025-03-14T07:49:27.7206352Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-a7d713f3ed49cf31.xml (deflated 90%) 2025-03-14T07:49:27.7207775Z adding: test/test-reports/python-pytest/distributed.tensor.experimental.test_tp_transform/distributed.tensor.experimental.test_tp_transform-9f8b7a419de45c32.xml (deflated 60%) 2025-03-14T07:49:27.7209207Z adding: test/test-reports/python-pytest/distributed.tensor.test_utils/distributed.tensor.test_utils-d9bece75845b3c48.xml (deflated 80%) 2025-03-14T07:49:27.7210651Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-52fc60cf89cd4d97.xml (deflated 63%) 2025-03-14T07:49:27.7212139Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-81826986ae4cc2e8.xml (deflated 79%) 2025-03-14T07:49:27.7213474Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-51eb0a568e33a364.xml (deflated 81%) 2025-03-14T07:49:27.7214892Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_hsdp_checkpoint/distributed.checkpoint.test_hsdp_checkpoint-7bfce4f60b3437f4.xml (deflated 68%) 2025-03-14T07:49:27.7216368Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-9fb8f3647dc6fdaa.xml (deflated 75%) 2025-03-14T07:49:27.7217620Z adding: test/test-reports/python-pytest/distributed.test_nccl/distributed.test_nccl-e7d9bc86506c2d1f.xml (deflated 80%) 2025-03-14T07:49:27.7218909Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_model_state/distributed.checkpoint.test_fsdp_model_state-e05320064de462e2.xml (deflated 53%) 2025-03-14T07:49:27.7220441Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_checkpoint/distributed.checkpoint.test_dtensor_checkpoint-966243047279abf4.xml (deflated 41%) 2025-03-14T07:49:27.7221959Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_planner/distributed.checkpoint.test_planner-864a60ffdfc7d3e4.xml (deflated 75%) 2025-03-14T07:49:27.7223430Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_resharding/distributed.checkpoint.test_dtensor_resharding-33420e53f1ce477a.xml (deflated 75%) 2025-03-14T07:49:27.7224871Z adding: test/test-reports/python-pytest/distributed.tensor.test_matrix_ops/distributed.tensor.test_matrix_ops-be35f47f6225b8ac.xml (deflated 78%) 2025-03-14T07:49:27.7226274Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_backward_prefetch/distributed.fsdp.test_fsdp_backward_prefetch-fc576c27cab81ff9.xml (deflated 40%) 2025-03-14T07:49:27.7227687Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_utils/distributed.checkpoint.test_utils-c9f999b47defefbf.xml (deflated 77%) 2025-03-14T07:49:27.7229120Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-e026d6fd05b4a537.xml (deflated 85%) 2025-03-14T07:49:27.7230695Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-6f68b438b235c497.xml (deflated 58%) 2025-03-14T07:49:27.7232424Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_init/distributed._composable.fsdp.test_fully_shard_init-24b33f7ab4480797.xml (deflated 87%) 2025-03-14T07:49:27.7234241Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_optim_state/distributed.fsdp.test_fsdp_optim_state-c103a55e57ace96a.xml (deflated 93%) 2025-03-14T07:49:27.7235754Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-822ec8e07ff3d196.xml (deflated 34%) 2025-03-14T07:49:27.7237541Z adding: test/test-reports/python-pytest/distributed._composable.test_composability.test_2d_composability/distributed._composable.test_composability.test_2d_composability-e3d869e89f36a86f.xml (deflated 83%) 2025-03-14T07:49:27.7239364Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_exec_order/distributed.fsdp.test_fsdp_exec_order-4a96587fe59da768.xml (deflated 81%) 2025-03-14T07:49:27.7240884Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-f8b1f4e3351257dd.xml (deflated 93%) 2025-03-14T07:49:27.7242565Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint/distributed.checkpoint.test_file_system_checkpoint-63e3bbe604822322.xml (deflated 79%) 2025-03-14T07:49:27.7244324Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_sharded_grad_scaler/distributed.fsdp.test_fsdp_sharded_grad_scaler-89aaa95cbd115302.xml (deflated 89%) 2025-03-14T07:49:27.7246303Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint_cpu/distributed.checkpoint.test_file_system_checkpoint_cpu-f48816ffa31e2d0c.xml (deflated 81%) 2025-03-14T07:49:27.7247859Z adding: test/test-reports/python-pytest/distributed.launcher.test_run/distributed.launcher.test_run-259bdc98795534eb.xml (deflated 84%) 2025-03-14T07:49:27.7249290Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_forward/distributed.fsdp.test_fsdp_multiple_forward-f78733cf560583b0.xml (deflated 28%) 2025-03-14T07:49:27.7250905Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_tensor_ops/distributed._shard.sharded_tensor.ops.test_tensor_ops-1abf1b0437579e30.xml (deflated 68%) 2025-03-14T07:49:27.7252594Z adding: test/test-reports/python-pytest/distributed._shard.sharded_optim.test_sharded_optim/distributed._shard.sharded_optim.test_sharded_optim-ae998032661cd86f.xml (deflated 52%) 2025-03-14T07:49:27.7254049Z adding: test/test-reports/python-pytest/distributed.test_launcher/distributed.test_launcher-b242183a4345e972.xml (deflated 40%) 2025-03-14T07:49:27.7255656Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-1b3cfcb6fbc858f1.xml (deflated 65%) 2025-03-14T07:49:27.7257336Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_init/distributed._shard.sharded_tensor.ops.test_init-40fdf2ceb867b91a.xml (deflated 61%) 2025-03-14T07:49:27.7258800Z adding: test/test-reports/python-pytest/distributed._shard.test_sharder/distributed._shard.test_sharder-1af0b2aebeb7d108.xml (deflated 51%) 2025-03-14T07:49:27.7260083Z adding: test/test-reports/python-pytest/distributed.fsdp.test_wrap/distributed.fsdp.test_wrap-814171d6f20f7c78.xml (deflated 85%) 2025-03-14T07:49:27.7261540Z adding: test/test-reports/python-pytest/distributed._shard.sharding_spec.test_sharding_spec/distributed._shard.sharding_spec.test_sharding_spec-70e33a21e0a1d76b.xml (deflated 81%) 2025-03-14T07:49:27.7263130Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_use_orig_params/distributed.fsdp.test_fsdp_use_orig_params-40f4c746b37896c9.xml (deflated 85%) 2025-03-14T07:49:27.7264534Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-00c88cc1bda239f2.xml (deflated 82%) 2025-03-14T07:49:27.7265876Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-cc96b6681e0952bd.xml (deflated 82%) 2025-03-14T07:49:27.7267259Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-9195e8ea91b1bcff.xml (deflated 82%) 2025-03-14T07:49:27.7268461Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-a495b36582133210.xml (deflated 82%) 2025-03-14T07:49:27.7270169Z adding: test/test-reports/python-pytest/distributed._tools.test_fake_collectives/distributed._tools.test_fake_collectives-6a15b285117b5bb3.xml (deflated 38%) 2025-03-14T07:49:27.7271674Z adding: test/test-reports/python-pytest/distributed.elastic.events.lib_test/distributed.elastic.events.lib_test-8de2aa4a6c99537f.xml (deflated 76%) 2025-03-14T07:49:27.7273238Z adding: test/test-reports/python-pytest/distributed.elastic.metrics.api_test/distributed.elastic.metrics.api_test-1ff81862e7e6a118.xml (deflated 60%) 2025-03-14T07:49:27.7275080Z adding: test/test-reports/python-pytest/distributed.elastic.multiprocessing.api_test/distributed.elastic.multiprocessing.api_test-32bff9aa1001bac1.xml (deflated 84%) 2025-03-14T07:49:27.7276827Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_example/distributed.elastic.timer.local_timer_example-eac34d684180ce08.xml (deflated 50%) 2025-03-14T07:49:27.7278515Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_test/distributed.elastic.timer.local_timer_test-b76b3eecf889754a.xml (deflated 80%) 2025-03-14T07:49:27.7280190Z adding: test/test-reports/python-pytest/distributed.elastic.utils.distributed_test/distributed.elastic.utils.distributed_test-3ea3b01989a6e6cf.xml (deflated 79%) 2025-03-14T07:49:27.7281846Z adding: test/test-reports/python-pytest/distributed.elastic.utils.logging_test/distributed.elastic.utils.logging_test-a4aef255961f516b.xml (deflated 51%) 2025-03-14T07:49:27.7283417Z adding: test/test-reports/python-pytest/distributed.elastic.utils.util_test/distributed.elastic.utils.util_test-553bbde35121afda.xml (deflated 81%) 2025-03-14T07:49:27.7284780Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-5cd143f5be00104a.xml (deflated 60%) 2025-03-14T07:49:27.7285830Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-6ba91ba33eb388ac.xml (deflated 60%) 2025-03-14T07:49:27.7286725Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-51f49187f74c33f3.xml (deflated 69%) 2025-03-14T07:49:27.7287615Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-ee4efc38a445a361.xml (deflated 78%) 2025-03-14T07:49:27.7288507Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-d83301cd0686e283.xml (deflated 74%) 2025-03-14T07:49:27.7289500Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-d703c6fe075629f5.xml (deflated 78%) 2025-03-14T07:49:27.7290400Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-4b3315072aafddcd.xml (deflated 64%) 2025-03-14T07:49:27.7291285Z adding: test/test-reports/python-pytest/test_linalg/test_linalg-46b50f2131b351b6.xml (deflated 43%) 2025-03-14T07:49:27.7318470Z ##[group]Run # Remove any previous usage logs if they exist 2025-03-14T07:49:27.7319024Z # Remove any previous usage logs if they exist 2025-03-14T07:49:27.7319476Z rm -f logs-*.zip 2025-03-14T07:49:27.7320044Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2025-03-14T07:49:27.7320704Z # so check to see if the file exists first 2025-03-14T07:49:27.7321141Z if [ -f 'usage_log.txt' ]; then 2025-03-14T07:49:27.7321596Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2025-03-14T07:49:27.7322021Z fi 2025-03-14T07:49:27.7322469Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2025-03-14T07:49:27.7323140Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2025-03-14T07:49:27.7323618Z fi 2025-03-14T07:49:27.7329202Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:27.7329611Z env: 2025-03-14T07:49:27.7329849Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:27.7330212Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:27.7330800Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:27.7331523Z FILE_SUFFIX: test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799 2025-03-14T07:49:27.7332037Z ##[endgroup] 2025-03-14T07:49:27.7505259Z adding: usage_log.txt (deflated 97%) 2025-03-14T07:49:27.7584985Z adding: test/test-reports/distributed.test_distributed_spawn_1.10_871ab4422f46658d_.log (deflated 12%) 2025-03-14T07:49:27.7586154Z adding: test/test-reports/distributed.test_distributed_spawn_1.10_e5fc3e94f7bc89d6_.log (deflated 12%) 2025-03-14T07:49:27.7591533Z adding: test/test-reports/distributed.test_distributed_spawn_1.10_fcce9f74fd866ffb_.log (deflated 94%) 2025-03-14T07:49:27.7598081Z adding: test/test-reports/distributed.test_distributed_spawn_1.10_a98da89a2b5bf43e_.log (deflated 94%) 2025-03-14T07:49:27.7604834Z adding: test/test-reports/distributed.test_distributed_spawn_1.10_06c84ed7b4e19b0f_.log (deflated 94%) 2025-03-14T07:49:27.7611348Z adding: test/test-reports/distributed.test_distributed_spawn_1.10_f0cafc9ee0c3eb5e_.log (deflated 94%) 2025-03-14T07:49:27.7612481Z adding: test/test-reports/distributed.test_distributed_spawn_4.10_2e2778bf3ed098b4_.log (deflated 12%) 2025-03-14T07:49:27.7613437Z adding: test/test-reports/distributed.test_distributed_spawn_4.10_3492154bb7e0a134_.log (deflated 12%) 2025-03-14T07:49:27.7616495Z adding: test/test-reports/distributed.test_distributed_spawn_4.10_7f7b549a1ba83a50_.log (deflated 94%) 2025-03-14T07:49:27.7620933Z adding: test/test-reports/distributed.test_distributed_spawn_4.10_d880a192cb7a84f7_.log (deflated 94%) 2025-03-14T07:49:27.7625549Z adding: test/test-reports/distributed.test_distributed_spawn_4.10_1208f7a57f541361_.log (deflated 94%) 2025-03-14T07:49:27.7629936Z adding: test/test-reports/distributed.test_distributed_spawn_4.10_df216ba9002c4717_.log (deflated 94%) 2025-03-14T07:49:27.7631065Z adding: test/test-reports/distributed.test_distributed_spawn_7.10_e2a4499c0bedac18_.log (deflated 12%) 2025-03-14T07:49:27.7632024Z adding: test/test-reports/distributed.test_distributed_spawn_7.10_c70845217ecb2c87_.log (deflated 12%) 2025-03-14T07:49:27.7635762Z adding: test/test-reports/distributed.test_distributed_spawn_7.10_e3ad0f310aa2c794_.log (deflated 94%) 2025-03-14T07:49:27.7640695Z adding: test/test-reports/distributed.test_distributed_spawn_7.10_cac41c17512933c8_.log (deflated 94%) 2025-03-14T07:49:27.7645913Z adding: test/test-reports/distributed.test_distributed_spawn_7.10_6abc9d64bda0e7a9_.log (deflated 94%) 2025-03-14T07:49:27.7650598Z adding: test/test-reports/distributed.test_distributed_spawn_7.10_b01615d06992488a_.log (deflated 94%) 2025-03-14T07:49:27.7651732Z adding: test/test-reports/distributed.test_distributed_spawn_10.10_e9357b001ab413d6_.log (deflated 12%) 2025-03-14T07:49:27.7652700Z adding: test/test-reports/distributed.test_distributed_spawn_10.10_705d3080ac729201_.log (deflated 12%) 2025-03-14T07:49:27.7655899Z adding: test/test-reports/distributed.test_distributed_spawn_10.10_2e9557be05776e88_.log (deflated 92%) 2025-03-14T07:49:27.7660515Z adding: test/test-reports/distributed.test_distributed_spawn_10.10_36d9f47aec036e6a_.log (deflated 92%) 2025-03-14T07:49:27.7665449Z adding: test/test-reports/distributed.test_distributed_spawn_10.10_f78ab49fec7a5991_.log (deflated 92%) 2025-03-14T07:49:27.7670489Z adding: test/test-reports/distributed.test_distributed_spawn_10.10_aedef97445abce8f_.log (deflated 92%) 2025-03-14T07:49:27.7724731Z adding: test/test-reports/distributed.fsdp.test_fsdp_core_3.3_1e13e702cce053f8_.log (deflated 97%) 2025-03-14T07:49:27.7725900Z adding: test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_bde77c173878fd45_.log (deflated 65%) 2025-03-14T07:49:27.7727885Z adding: test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_7bba5eef084b80f0_.log (deflated 86%) 2025-03-14T07:49:27.7743372Z adding: test/test-reports/distributed.test_c10d_nccl_1.3_04f54c1768469a48_.log (deflated 93%) 2025-03-14T07:49:27.7744522Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_5023abcb7f83d749_.log (deflated 86%) 2025-03-14T07:49:27.7745724Z adding: test/test-reports/distributed.test_c10d_pypg_1.1_75010ceb3efbbbb1_.log (deflated 87%) 2025-03-14T07:49:27.7746653Z adding: test/test-reports/distributed.test_c10d_logger_1.1_c6885017d23a1e2f_.log (deflated 67%) 2025-03-14T07:49:27.7747625Z adding: test/test-reports/distributed.test_c10d_object_collectives_1.1_8cd8a722fe67e290_.log (deflated 76%) 2025-03-14T07:49:27.7758629Z adding: test/test-reports/distributed.test_c10d_gloo_1.2_1713d75f1c19317b_.log (deflated 95%) 2025-03-14T07:49:27.7761643Z adding: test/test-reports/distributed.tensor.test_attention_1.1_cecac96004c5a704_.log (deflated 93%) 2025-03-14T07:49:27.7762665Z adding: test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_b472f4ff64767c22_.log (deflated 69%) 2025-03-14T07:49:27.7763655Z adding: test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_0fe9c845225922a4_.log (deflated 53%) 2025-03-14T07:49:27.7764639Z adding: test/test-reports/distributed.tensor.test_experimental_ops_1.1_63313631dc501593_.log (deflated 61%) 2025-03-14T07:49:27.7765773Z adding: test/test-reports/distributed.elastic.utils.util_test_1.1_7f56f554d46c499e_.log (deflated 75%) 2025-03-14T07:49:27.7766856Z adding: test/test-reports/distributed.tensor.experimental.test_register_sharding_1.1_e07afc1483098d43_.log (deflated 60%) 2025-03-14T07:49:27.7767890Z adding: test/test-reports/distributed.test_backends_1.1_66b0ef58f46d37e6_.log (deflated 57%) 2025-03-14T07:49:27.7768848Z adding: test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_e7fb4bbd490746d5_.log (deflated 91%) 2025-03-14T07:49:27.7769979Z adding: test/test-reports/distributed.tensor.test_init_1.1_ac30809b44654a27_.log (deflated 88%) 2025-03-14T07:49:27.7770814Z adding: test/test-reports/cpp.test_cpp_rpc_1.1_468aabb211f7f9db_.log (deflated 78%) 2025-03-14T07:49:27.7773305Z adding: test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_bd173762af82a151_.log (deflated 95%) 2025-03-14T07:49:27.7774359Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_3086fafbb295b4fd_.log (deflated 57%) 2025-03-14T07:49:27.7775371Z adding: test/test-reports/distributed._tools.test_memory_tracker_1.1_be482d49d172595f_.log (deflated 54%) 2025-03-14T07:49:27.7776499Z adding: test/test-reports/distributed.test_collective_utils_1.1_08c3f9d244e6b6e4_.log (stored 0%) 2025-03-14T07:49:27.7777437Z adding: test/test-reports/distributed.tensor.test_random_ops_1.1_09b22c5ff8f303d5_.log (deflated 80%) 2025-03-14T07:49:27.7778416Z adding: test/test-reports/distributed.pipelining.test_backward_1.1_99de1fa652728e8e_.log (deflated 67%) 2025-03-14T07:49:27.7779394Z adding: test/test-reports/distributed.pipelining.test_pipe_1.1_6997959cab6d6475_.log (deflated 61%) 2025-03-14T07:49:27.7780338Z adding: test/test-reports/distributed.test_device_mesh_1.1_5b875dc34bd2af4f_.log (deflated 89%) 2025-03-14T07:49:27.7781789Z adding: test/test-reports/distributed.tensor.experimental.test_tp_transform_1.1_264dd061635863c2_.log (deflated 93%) 2025-03-14T07:49:27.7782938Z adding: test/test-reports/distributed.tensor.test_utils_1.1_ef6812c1a5262015_.log (deflated 84%) 2025-03-14T07:49:27.7783967Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_9b1a1f4fb8bbd663_.log (deflated 58%) 2025-03-14T07:49:27.7785026Z adding: test/test-reports/distributed.pipelining.test_stage_1.1_d72c690ccddfb8de_.log (deflated 81%) 2025-03-14T07:49:27.7786052Z adding: test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_0acc1cfa96819b4c_.log (deflated 92%) 2025-03-14T07:49:27.7787440Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_731181cab7811b7d_.log (deflated 85%) 2025-03-14T07:49:27.7788364Z adding: test/test-reports/distributed.test_nccl_1.1_4cc3de62eab05fdb_.log (deflated 74%) 2025-03-14T07:49:27.7790449Z adding: test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_1a609f2b1c8dfecd_.log (deflated 94%) 2025-03-14T07:49:27.7791527Z adding: test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_d72c52315128d20f_.log (deflated 55%) 2025-03-14T07:49:27.7792571Z adding: test/test-reports/distributed.checkpoint.test_planner_1.1_271fc8d04a72ebc3_.log (deflated 74%) 2025-03-14T07:49:27.7793891Z adding: test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_990931a7831f3158_.log (deflated 91%) 2025-03-14T07:49:27.7794965Z adding: test/test-reports/distributed.optim.test_named_optimizer_1.1_b5aca0775e2c614d_.log (stored 0%) 2025-03-14T07:49:27.7795962Z adding: test/test-reports/distributed.tensor.test_matrix_ops_1.1_27832d6bdeca10c8_.log (deflated 75%) 2025-03-14T07:49:27.7796998Z adding: test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_e83714b1c2e5c064_.log (deflated 66%) 2025-03-14T07:49:27.7798037Z adding: test/test-reports/distributed.checkpoint.test_utils_1.1_e71461eaeb7a068a_.log (deflated 84%) 2025-03-14T07:49:27.7799082Z adding: test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_6e9c488205a6bcf6_.log (deflated 83%) 2025-03-14T07:49:27.7800039Z adding: test/test-reports/cpp.FileStoreTest_1.1_ea6c756c02b54124_.log (deflated 71%) 2025-03-14T07:49:27.7801031Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_adb77f0a783078dc_.log (deflated 83%) 2025-03-14T07:49:27.7802201Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_init_1.1_e32f372dd6e0eb52_.log (deflated 87%) 2025-03-14T07:49:27.7839697Z adding: test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_e8d2fb904d823585_.log (deflated 98%) 2025-03-14T07:49:27.7840752Z adding: test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_acd21a682c1b04bb_.log (deflated 55%) 2025-03-14T07:49:27.7843136Z adding: test/test-reports/distributed._composable.test_composability.test_2d_composability_1.1_66b549208c08cd35_.log (deflated 92%) 2025-03-14T07:49:27.7844538Z adding: test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_add41dca532558d3_.log (deflated 93%) 2025-03-14T07:49:27.7849886Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_89dabd6e3944091b_.log (deflated 89%) 2025-03-14T07:49:27.7852494Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_20d2e084dd03c39d_.log (deflated 93%) 2025-03-14T07:49:27.7855645Z adding: test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_20f16d6dc20e5f3c_.log (deflated 96%) 2025-03-14T07:49:27.7858769Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_3358339f4e23f51d_.log (deflated 94%) 2025-03-14T07:49:27.7861516Z adding: test/test-reports/distributed.launcher.test_run_1.1_2a36a401f8b2bd2d_.log (deflated 89%) 2025-03-14T07:49:27.7862672Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_2ea4316eade43f52_.log (deflated 52%) 2025-03-14T07:49:27.7863695Z adding: test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_b24dd91790bb736c_.log (deflated 77%) 2025-03-14T07:49:27.7864925Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_c10d84e8b0942997_.log (deflated 92%) 2025-03-14T07:49:27.7866728Z adding: test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_f2e9eedbf3fda571_.log (deflated 92%) 2025-03-14T07:49:27.7867751Z adding: test/test-reports/distributed.test_launcher_1.1_325ca958c68f67b8_.log (deflated 59%) 2025-03-14T07:49:27.7883218Z adding: test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_ec8fc303519d7cb3_.log (deflated 97%) 2025-03-14T07:49:27.7884779Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_373d7a5d7289d088_.log (deflated 89%) 2025-03-14T07:49:27.7886382Z adding: test/test-reports/distributed._shard.test_sharder_1.1_1e5aa8ced0a1795c_.log (deflated 88%) 2025-03-14T07:49:27.7890338Z adding: test/test-reports/distributed.fsdp.test_wrap_1.1_9306b2c7b3faa3f7_.log (deflated 94%) 2025-03-14T07:49:27.7891505Z adding: test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_a145ae4b1a3d8d56_.log (stored 0%) 2025-03-14T07:49:27.7892614Z adding: test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_58e258755e75201a_.log (deflated 84%) 2025-03-14T07:49:27.7915397Z adding: test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_fdcdec8a8426b810_.log (deflated 96%) 2025-03-14T07:49:27.7917487Z adding: test/test-reports/distributed.test_c10d_ops_nccl_1.1_4084e5ba55b71056_.log (deflated 93%) 2025-03-14T07:49:27.7918493Z adding: test/test-reports/distributed._tools.test_fake_collectives_1.1_0ee0a27d47da00af_.log (deflated 55%) 2025-03-14T07:49:27.7919542Z adding: test/test-reports/distributed.elastic.events.lib_test_1.1_1193088999bba53b_.log (deflated 72%) 2025-03-14T07:49:27.7920563Z adding: test/test-reports/distributed.elastic.metrics.api_test_1.1_501c26aa3a3a83fd_.log (deflated 62%) 2025-03-14T07:49:27.7921926Z adding: test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_d8bc2db5aaf0aedc_.log (deflated 84%) 2025-03-14T07:49:27.7923102Z adding: test/test-reports/distributed.elastic.timer.local_timer_example_1.1_10945b9933129ac0_.log (deflated 68%) 2025-03-14T07:49:27.7924216Z adding: test/test-reports/distributed.elastic.timer.local_timer_test_1.1_d691d350873f01b0_.log (deflated 77%) 2025-03-14T07:49:27.7926158Z adding: test/test-reports/distributed.elastic.utils.distributed_test_1.1_d959277c8c24e587_.log (deflated 93%) 2025-03-14T07:49:27.7927209Z adding: test/test-reports/distributed.elastic.utils.logging_test_1.1_52d00981044b3469_.log (deflated 58%) 2025-03-14T07:49:27.7928115Z adding: test/test-reports/cpp.HashStoreTest_1.1_d321ea168b05577d_.log (deflated 71%) 2025-03-14T07:49:27.7928920Z adding: test/test-reports/cpp.TCPStoreTest_1.1_e57b4d8c21b7ef55_.log (deflated 78%) 2025-03-14T07:49:27.7929687Z adding: test/test-reports/test_linalg_1.1_a8d367a0c85dc6f3_.log (deflated 49%) 2025-03-14T07:49:27.7930507Z adding: test/test-reports/cpp.ProcessGroupGlooTest_1.1_2aac4f1dfd96abaa_.log (deflated 84%) 2025-03-14T07:49:27.7931398Z adding: test/test-reports/cpp.ProcessGroupNCCLTest_1.1_7b66f4037c17f7ef_.log (deflated 82%) 2025-03-14T07:49:27.7962375Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-03-14T07:49:27.7963011Z # Remove any previous debugging artifacts if they exist 2025-03-14T07:49:27.7963628Z rm -f debug-*.zip 2025-03-14T07:49:27.7963982Z if [ -d 'test/debug' ]; then 2025-03-14T07:49:27.7964420Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-03-14T07:49:27.7964831Z fi 2025-03-14T07:49:27.7971093Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:27.7971542Z env: 2025-03-14T07:49:27.7971793Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:27.7972187Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:27.7972827Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:27.7973613Z FILE_SUFFIX: test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799 2025-03-14T07:49:27.7974173Z ##[endgroup] 2025-03-14T07:49:27.8071602Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-03-14T07:49:27.8071999Z with: 2025-03-14T07:49:27.8072261Z s3-bucket: gha-artifacts 2025-03-14T07:49:27.8072658Z s3-prefix: pytorch/pytorch/13849515290/1/artifact 2025-03-14T07:49:27.8073148Z retention-days: 14 2025-03-14T07:49:27.8073629Z if-no-files-found: warn 2025-03-14T07:49:27.8073968Z path: test-jsons-*.zip 2025-03-14T07:49:27.8074285Z name: artifact 2025-03-14T07:49:27.8074567Z region: us-east-1 2025-03-14T07:49:27.8074837Z env: 2025-03-14T07:49:27.8075100Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:27.8075625Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:27.8076291Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:27.8076882Z ##[endgroup] 2025-03-14T07:49:28.1771296Z NOTE: s3-prefix specified, ignoring name parameter 2025-03-14T07:49:28.1771872Z With the provided path, there will be 1 file uploaded 2025-03-14T07:49:28.1772418Z Uploading to s3 prefix: pytorch/pytorch/13849515290/1/artifact 2025-03-14T07:49:28.1815725Z Starting upload of test-jsons-test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799.zip 2025-03-14T07:49:28.3681776Z Finished upload of test-jsons-test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799.zip 2025-03-14T07:49:28.3861030Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-03-14T07:49:28.3861419Z with: 2025-03-14T07:49:28.3861678Z s3-bucket: gha-artifacts 2025-03-14T07:49:28.3862043Z s3-prefix: pytorch/pytorch/13849515290/1/artifact 2025-03-14T07:49:28.3862450Z retention-days: 14 2025-03-14T07:49:28.3862744Z if-no-files-found: error 2025-03-14T07:49:28.3863058Z path: test-reports-*.zip 2025-03-14T07:49:28.3863359Z name: artifact 2025-03-14T07:49:28.3863611Z region: us-east-1 2025-03-14T07:49:28.3864043Z env: 2025-03-14T07:49:28.3864301Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:28.3864693Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:28.3865328Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:28.3865891Z ##[endgroup] 2025-03-14T07:49:28.7297368Z NOTE: s3-prefix specified, ignoring name parameter 2025-03-14T07:49:28.7298430Z With the provided path, there will be 1 file uploaded 2025-03-14T07:49:28.7299031Z Uploading to s3 prefix: pytorch/pytorch/13849515290/1/artifact 2025-03-14T07:49:28.7340024Z Starting upload of test-reports-test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799.zip 2025-03-14T07:49:28.9403123Z Finished upload of test-reports-test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799.zip 2025-03-14T07:49:28.9588401Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-03-14T07:49:28.9588790Z with: 2025-03-14T07:49:28.9589059Z s3-bucket: gha-artifacts 2025-03-14T07:49:28.9589433Z s3-prefix: pytorch/pytorch/13849515290/1/artifact 2025-03-14T07:49:28.9589840Z retention-days: 14 2025-03-14T07:49:28.9590145Z if-no-files-found: ignore 2025-03-14T07:49:28.9590464Z path: logs-*.zip 2025-03-14T07:49:28.9590749Z name: artifact 2025-03-14T07:49:28.9591026Z region: us-east-1 2025-03-14T07:49:28.9591294Z env: 2025-03-14T07:49:28.9591716Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:28.9592102Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:28.9592714Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:28.9593557Z ##[endgroup] 2025-03-14T07:49:29.3011085Z NOTE: s3-prefix specified, ignoring name parameter 2025-03-14T07:49:29.3011796Z With the provided path, there will be 1 file uploaded 2025-03-14T07:49:29.3012358Z Uploading to s3 prefix: pytorch/pytorch/13849515290/1/artifact 2025-03-14T07:49:29.3053454Z Starting upload of logs-test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799.zip 2025-03-14T07:49:29.4979509Z Finished upload of logs-test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799.zip 2025-03-14T07:49:29.5158202Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-03-14T07:49:29.5174957Z with: 2025-03-14T07:49:29.5175389Z s3-bucket: gha-artifacts 2025-03-14T07:49:29.5175808Z s3-prefix: pytorch/pytorch/13849515290/1/artifact 2025-03-14T07:49:29.5176272Z retention-days: 14 2025-03-14T07:49:29.5176598Z if-no-files-found: ignore 2025-03-14T07:49:29.5176940Z path: debug-*.zip 2025-03-14T07:49:29.5177239Z name: artifact 2025-03-14T07:49:29.5177515Z region: us-east-1 2025-03-14T07:49:29.5177789Z env: 2025-03-14T07:49:29.5178059Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:29.5178686Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:29.5179352Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:29.5179950Z ##[endgroup] 2025-03-14T07:49:29.8558181Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-03-14T07:49:29.8759046Z ##[group]Run # shellcheck disable=SC2156 2025-03-14T07:49:29.8759509Z # shellcheck disable=SC2156 2025-03-14T07:49:29.8760225Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-03-14T07:49:29.8766855Z shell: /usr/bin/bash -e {0} 2025-03-14T07:49:29.8767192Z env: 2025-03-14T07:49:29.8767467Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:29.8767874Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:29.8768530Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:29.8769115Z ##[endgroup] 2025-03-14T07:49:30.1331503Z Prepare all required actions 2025-03-14T07:49:30.1331945Z Getting action download info 2025-03-14T07:49:30.2487442Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-03-14T07:49:30.2487847Z with: 2025-03-14T07:49:30.2488101Z job_id: 38755323799 2025-03-14T07:49:30.2488664Z job_name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T07:49:30.2489282Z workflow_name: pull 2025-03-14T07:49:30.2489575Z workflow_run_id: 13849515290 2025-03-14T07:49:30.2489891Z workflow_attempt: 1 2025-03-14T07:49:30.2490169Z env: 2025-03-14T07:49:30.2490417Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:30.2490814Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:30.2491418Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:30.2491957Z ##[endgroup] 2025-03-14T07:49:30.2514498Z ##[group]Run echo "workflow_id: 13849515290" 2025-03-14T07:49:30.2515037Z echo "workflow_id: 13849515290" 2025-03-14T07:49:30.2515456Z echo "workflow_attempt: 1" 2025-03-14T07:49:30.2515860Z echo "workflow_Name: pull" 2025-03-14T07:49:30.2516253Z echo "job_id: 38755323799" 2025-03-14T07:49:30.2516992Z echo "job_name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu)" 2025-03-14T07:49:30.2523614Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:30.2524087Z env: 2025-03-14T07:49:30.2524375Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:30.2524799Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:30.2525769Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:30.2526310Z ##[endgroup] 2025-03-14T07:49:30.2549974Z workflow_id: 13849515290 2025-03-14T07:49:30.2550370Z workflow_attempt: 1 2025-03-14T07:49:30.2550682Z workflow_Name: pull 2025-03-14T07:49:30.2551025Z job_id: 38755323799 2025-03-14T07:49:30.2551664Z job_name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) 2025-03-14T07:49:30.2595473Z ##[group]Run nick-fields/retry@v3.0.0 2025-03-14T07:49:30.2595859Z with: 2025-03-14T07:49:30.2596126Z shell: bash 2025-03-14T07:49:30.2596411Z timeout_minutes: 5 2025-03-14T07:49:30.2596724Z max_attempts: 5 2025-03-14T07:49:30.2597033Z retry_wait_seconds: 30 2025-03-14T07:49:30.2597611Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 2025-03-14T07:49:30.2598246Z polling_interval_seconds: 1 2025-03-14T07:49:30.2598607Z warning_on_retry: true 2025-03-14T07:49:30.2598958Z continue_on_error: false 2025-03-14T07:49:30.2599288Z env: 2025-03-14T07:49:30.2599561Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:30.2599975Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:30.2600651Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:30.2601259Z ##[endgroup] 2025-03-14T07:49:30.6085095Z Defaulting to user installation because normal site-packages is not writeable 2025-03-14T07:49:30.6256419Z Requirement already satisfied: python-dateutil==2.8.2 in /home/ec2-user/.local/lib/python3.9/site-packages (2.8.2) 2025-03-14T07:49:31.6796759Z Collecting boto3==1.35.42 2025-03-14T07:49:31.6823445Z Using cached boto3-1.35.42-py3-none-any.whl (139 kB) 2025-03-14T07:49:31.6850844Z Requirement already satisfied: pandas==2.1.3 in /home/ec2-user/.local/lib/python3.9/site-packages (2.1.3) 2025-03-14T07:49:31.6866961Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0) 2025-03-14T07:49:31.6914671Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-03-14T07:49:31.6916763Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-03-14T07:49:31.6921512Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-03-14T07:49:31.7797925Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1) 2025-03-14T07:49:31.7800807Z Requirement already satisfied: tzdata>=2022.1 in /home/ec2-user/.local/lib/python3.9/site-packages (from pandas==2.1.3) (2025.1) 2025-03-14T07:49:31.7815681Z Requirement already satisfied: numpy<2,>=1.22.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from pandas==2.1.3) (1.26.4) 2025-03-14T07:49:31.7876051Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-03-14T07:49:31.8760047Z Installing collected packages: boto3 2025-03-14T07:49:31.8760514Z Attempting uninstall: boto3 2025-03-14T07:49:31.8763394Z Found existing installation: boto3 1.35.33 2025-03-14T07:49:31.8869673Z Uninstalling boto3-1.35.33: 2025-03-14T07:49:31.8881423Z Successfully uninstalled boto3-1.35.33 2025-03-14T07:49:31.9454742Z Successfully installed boto3-1.35.42 2025-03-14T07:49:32.3417855Z Command completed after 1 attempt(s). 2025-03-14T07:49:32.3484114Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-03-14T07:49:32.3484899Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-03-14T07:49:32.3485482Z  --workflow-run-id "13849515290" \ 2025-03-14T07:49:32.3485891Z  --workflow-name "pull" \ 2025-03-14T07:49:32.3486283Z  --workflow-run-attempt "1" \ 2025-03-14T07:49:32.3486889Z  --job-id "38755323799" \ 2025-03-14T07:49:32.3487556Z  --job-name "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu)" 2025-03-14T07:49:32.3494243Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:32.3494714Z env: 2025-03-14T07:49:32.3494992Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:32.3495395Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:32.3496020Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:32.3496583Z ##[endgroup] 2025-03-14T07:49:38.0560377Z repo: pytorch/pytorch 2025-03-14T07:49:38.0561112Z Downloading logs-test-distributed-1-3-lf.linux.g4dn.12xlarge.nvidia.gpu_38755323799.zip 2025-03-14T07:49:38.0561829Z Converted Log Model: UtilizationMetadata: 2025-03-14T07:49:38.0563579Z UtilizationMetadata(level='metadata', workflow_id='13849515290', job_id='38755323799', workflow_name='pull', job_name='linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu)', usage_collect_interval=1.0, data_model_version=1.0, start_at=1741927066, gpu_count=4, cpu_count=48, gpu_type='pynvml', error=None) 2025-03-14T07:49:38.0565425Z [Db Segments] detected pytest cmd: 16, generated segments: 16 2025-03-14T07:49:38.0566038Z [db model] Peek db timeseries 2025-03-14T07:49:38.0566376Z :{ 2025-03-14T07:49:38.0566637Z "created_at": 1741938576, 2025-03-14T07:49:38.0566977Z "type": "utilization", 2025-03-14T07:49:38.0567294Z "tags": [ 2025-03-14T07:49:38.0567561Z "record" 2025-03-14T07:49:38.0567833Z ], 2025-03-14T07:49:38.0568095Z "time_stamp": 1741927066, 2025-03-14T07:49:38.0568438Z "repo": "pytorch/pytorch", 2025-03-14T07:49:38.0568782Z "workflow_id": 13849515290, 2025-03-14T07:49:38.0569118Z "run_attempt": 1, 2025-03-14T07:49:38.0569624Z "job_id": 38755323799, 2025-03-14T07:49:38.0569955Z "workflow_name": "pull", 2025-03-14T07:49:38.0570611Z "job_name": "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu)", 2025-03-14T07:49:38.0571274Z "json_data": "{}" 2025-03-14T07:49:38.0571565Z } 2025-03-14T07:49:38.0572535Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.0/pytorch/pytorch/13849515290/1/38755323799/metadata 2025-03-14T07:49:38.0573631Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.0/pytorch/pytorch/13849515290/1/38755323799/metadata 2025-03-14T07:49:38.0574760Z Writing 2296 documents to S3 ossci-utilization/util_timeseries/v_1.0/pytorch/pytorch/13849515290/1/38755323799/time_series 2025-03-14T07:49:38.0575914Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.0/pytorch/pytorch/13849515290/1/38755323799/time_series 2025-03-14T07:49:38.1562253Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-03-14T07:49:38.1562786Z with: 2025-03-14T07:49:38.1563045Z env: 2025-03-14T07:49:38.1563342Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:38.1563763Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:38.1564441Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:38.1565040Z ##[endgroup] 2025-03-14T07:49:38.1589440Z ##[group]Run set -eou pipefail 2025-03-14T07:49:38.1589797Z set -eou pipefail 2025-03-14T07:49:38.1590105Z  2025-03-14T07:49:38.1590514Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-03-14T07:49:38.1591022Z for _ in $(seq 1440); do 2025-03-14T07:49:38.1591390Z  # Break if no ssh session exists anymore 2025-03-14T07:49:38.1591779Z  if [ "$(who)" = "" ]; then 2025-03-14T07:49:38.1592111Z  break 2025-03-14T07:49:38.1592415Z  fi 2025-03-14T07:49:38.1592676Z  echo "." 2025-03-14T07:49:38.1593046Z  sleep 5 2025-03-14T07:49:38.1593489Z done 2025-03-14T07:49:38.1599904Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:38.1600375Z env: 2025-03-14T07:49:38.1600651Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:38.1601066Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:38.1601741Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:38.1602340Z ##[endgroup] 2025-03-14T07:49:38.1643691Z Holding runner for 2 hours until all ssh sessions have logged out 2025-03-14T07:49:38.1695188Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-03-14T07:49:38.1696025Z # ignore expansion of "docker ps -q" since it could be empty 2025-03-14T07:49:38.1696531Z # shellcheck disable=SC2046 2025-03-14T07:49:38.1696943Z docker stop $(docker ps -q) || true 2025-03-14T07:49:38.1697353Z # Prune all of the docker images 2025-03-14T07:49:38.1697864Z docker system prune -af 2025-03-14T07:49:38.1703229Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:38.1703646Z env: 2025-03-14T07:49:38.1703893Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:49:38.1704264Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:49:38.1704863Z DOCKER_CONTAINER_ID: 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:38.1705397Z ##[endgroup] 2025-03-14T07:49:38.5844033Z 1e25b2eec068 2025-03-14T07:49:39.0498893Z Deleted Containers: 2025-03-14T07:49:39.0499414Z 1e25b2eec0688f547d7484267126b6f695f2015b628b316e9bcdd68514b129b3 2025-03-14T07:49:39.0499801Z 2025-03-14T07:49:44.6976905Z Deleted Images: 2025-03-14T07:49:44.6977904Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T07:49:44.6979630Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:af572d83968aa426e5a9b40a69e9c34e3dd00ecdc49b56285f110770c0ab265b 2025-03-14T07:49:44.6980945Z deleted: sha256:f27528c1978e671c56c46509bcb1b201a995437aea125e51d1442a668bd40735 2025-03-14T07:49:44.6981723Z deleted: sha256:7df3b3d91e38faf68b7c313d3e22434851d96f581efc66a551c10a7b5fcb39e1 2025-03-14T07:49:44.6982509Z deleted: sha256:efe65fbd48105da05b145d32e9cfba188376feea6e3a4a9fee31c09910ab3f94 2025-03-14T07:49:44.6983290Z deleted: sha256:18e202e85aa24bed28e577e8aa689570f05fa2cedb144481b4e2523514975e59 2025-03-14T07:49:44.6984072Z deleted: sha256:c88fcb46aa81ce24cee8556178f6cd47f0018f5ebda0c447647cfaf9807b27d7 2025-03-14T07:49:44.6984853Z deleted: sha256:1b24be718155d5988c7eff41d12f9e47a59dc08d3dfe4f7f154ac0910f1e0e12 2025-03-14T07:49:44.6985639Z deleted: sha256:736efa0f7fdfc83c3a464c45e0edde2ea255f04b6b71e70dc03870678191ac57 2025-03-14T07:49:44.6986773Z deleted: sha256:a88d1310dd34af23ea003bdc60a2506c99169afbdb9d2abe84e5a760db76def9 2025-03-14T07:49:44.6987553Z deleted: sha256:6d0ef54a31461062b1a936112563b065a81c16da448d81952295a2c7c29ba5dc 2025-03-14T07:49:44.6988447Z deleted: sha256:727e022470f65d31a957e2d595fcaef88ad8c4494b3b22b30fa42fcf1d8e468f 2025-03-14T07:49:44.6989197Z deleted: sha256:5ac92ee63426be6001363e332f78c2bc4ef8d29cfa32c66971425ac845c8649f 2025-03-14T07:49:44.6989953Z deleted: sha256:ded1cb062fe826d5e80a3b9abf0be2a78033d169a9cf4a30c020209681ac9970 2025-03-14T07:49:44.6990710Z deleted: sha256:7515d02d435b1ccabd44441bab95d19625044dcaeea98e5223ba188185ed2fb6 2025-03-14T07:49:44.6991473Z deleted: sha256:bdb68e48af41b45aed6ac3d051f5c501f54ae4d9231943e262c540d56384ab6b 2025-03-14T07:49:44.6992227Z deleted: sha256:45c8d3c751ee55a238ca0b098407c50e138df6c69e6bb5bc8319db948a5abb4a 2025-03-14T07:49:44.6993053Z deleted: sha256:5bb7898a3c7d3fc26645871004326b899567ca0d65a287a2f02f58cca093a2d8 2025-03-14T07:49:44.6993991Z deleted: sha256:d2884fd5d42ae885d59b45ab84e3237e1e701190ec3a29bbf44613a3324494d0 2025-03-14T07:49:44.6994759Z deleted: sha256:134463c713e99e1c08204ad8e6a7ceee77895185b54bc5a3f11db19bb756838c 2025-03-14T07:49:44.6995531Z deleted: sha256:ae25edea2185381cf78fd23d7199c618526cbe110282dff5d335461d7de3632e 2025-03-14T07:49:44.6996450Z deleted: sha256:52a9419fe99336acb87232c84aea25c044628463b0be1aa8d83cc5f1069a6467 2025-03-14T07:49:44.6997222Z deleted: sha256:356d9f2dc3c370520d90afd5a6d301703105d23663d2ece739e5e87a8a87a6b8 2025-03-14T07:49:44.6997987Z deleted: sha256:c0417c483f0d26075071693c3a20898018d724f78527eaed9a08e5cab6f430ff 2025-03-14T07:49:44.6998759Z deleted: sha256:a8ae60d89f1aec1be3c0f3fb915343c583c6d17f623e67e1ad256f2d723b78de 2025-03-14T07:49:44.6999532Z deleted: sha256:504e1a0d2d9d4de944048777126440752388da2d8b9757e16cec166008636893 2025-03-14T07:49:44.7000303Z deleted: sha256:a8fd28f986bdb350db618bae49b370a3537ce20f1addc3aba93c0f4edb78878e 2025-03-14T07:49:44.7001102Z deleted: sha256:ed19a05ade4061fc3eb537091ecf97e5e867e6bbd7134bfbbe919a3371b8b6ad 2025-03-14T07:49:44.7001888Z deleted: sha256:1076a54312f9db9a32766c98dc226bc09892916e4020adcd86f89da5d16a27c5 2025-03-14T07:49:44.7002660Z deleted: sha256:5d6d164541c3646cfae0d5933eaac152ec6aa9649d4c8f4f1b84204ae25dd9fc 2025-03-14T07:49:44.7003450Z deleted: sha256:7cad81129c25005d549dfd39abf405f60babfa760fb1f7dc41909c13bd47aa43 2025-03-14T07:49:44.7004233Z deleted: sha256:e6ed6e34cf5a04ba7c7ed654190b455acb1da3352a5305e418932ca91afa86a7 2025-03-14T07:49:44.7005016Z deleted: sha256:ab0c6dd89a86d0762e0ca2201106adfa7043402dbfa7ff3632128dcbc8044f63 2025-03-14T07:49:44.7005889Z deleted: sha256:8a5f0096e2eae932366563a149961c402eb111c1e033e76b5f35bcf6d4920717 2025-03-14T07:49:44.7006655Z deleted: sha256:242eec06b8c94c33500cc13a6e4d0bc8abb33178f8352426f4363c5ee33b2453 2025-03-14T07:49:44.7007418Z deleted: sha256:086659ffa03d2ce1cac6801ddb3be6c0e87e5c8e2cd2a6dcf694c930060f9ab3 2025-03-14T07:49:44.7008171Z deleted: sha256:18635af34917c970f5131c6d23e20f4c5536e3129cec4048d484a68c52dba7f0 2025-03-14T07:49:44.7008915Z deleted: sha256:88f16201557466d77dd5da688eccf3b6f1556449beed2a272f7e735c2badd3f2 2025-03-14T07:49:44.7009668Z deleted: sha256:8a1e82e7ef5884f00b60a6ee4a823031a8867cc7a852aef12d3f06ce2c465219 2025-03-14T07:49:44.7010426Z deleted: sha256:56601d3684823ea1dbabd7b6204e67780ef9cf21020481ebfaccf9a61a25eebf 2025-03-14T07:49:44.7011190Z deleted: sha256:e426ebcdc41f28ad6618ab227381eda7f6e7abc6df607a9c3a558d86c9407289 2025-03-14T07:49:44.7011947Z deleted: sha256:5c6c7f4de56388447e39d455bd1f7de498ee3d8399076f02f323084043369716 2025-03-14T07:49:44.7012697Z deleted: sha256:262083418adf90b9314bff128229a3d85f163a0ab8c4b7cfa22feeb0d85ca110 2025-03-14T07:49:44.7013459Z deleted: sha256:3c638ab5c249ec1d781fcd14d0131b43b7bc9dc0b78ac55ca038f5cc42dbf561 2025-03-14T07:49:44.7014225Z deleted: sha256:da24f4cc75e46296cfc32d2217ebaf66b73632c1fe0497d2c477a3342dbaaffa 2025-03-14T07:49:44.7015030Z deleted: sha256:d73773609f3bd0ccd3703f6edc2c8705702f975e88eae0a2a21e8d44d13ff76b 2025-03-14T07:49:44.7015901Z deleted: sha256:dd54fd8af4f61bed798808f5cff1826feb159e9524b07d57ceb130ba39a2d48b 2025-03-14T07:49:44.7016661Z deleted: sha256:48770aa8456d18622b129708dc37003cea3a78fca9907e52f31ab9d53458be5d 2025-03-14T07:49:44.7017530Z deleted: sha256:d448c69507ad2f83268763519e92e549c8cf3b95e1f69152b4e98caf03044522 2025-03-14T07:49:44.7018347Z deleted: sha256:3cb17e9f20cd7ddcbdad4f7a175aff0f765cc67257edcb181ba34afc78fff8a4 2025-03-14T07:49:44.7019124Z deleted: sha256:d44205ee863fe3c4d6de98b39da0bb7ed387b2f53c2a2fafacf7c360b0a02a6c 2025-03-14T07:49:44.7019896Z deleted: sha256:b196c45e6fdcd754ac9ace7737f8faa0d6124e0f58dd3eb5a9b05c2cefea7dce 2025-03-14T07:49:44.7020661Z deleted: sha256:fae7242e90ba780de27c69e03c75f62eba295dc8e11dac4d8e14914e18158407 2025-03-14T07:49:44.7021423Z deleted: sha256:9eedcb7f3304a9b8b0a984671fa4644bfde50d5aff9f4881102d6b9f71647f8f 2025-03-14T07:49:44.7022177Z deleted: sha256:8005852aecf045af417dfb919a9a26dc530a38c8ced053243520c4d938084f1d 2025-03-14T07:49:44.7022934Z deleted: sha256:79a0e26b5eedf451682609635cf85aac9698cc6a6e4234ec6f2f8e8a7539d5b6 2025-03-14T07:49:44.7023704Z deleted: sha256:5aaffdb2acd18924cb4daa2ec1bfdc9e0167974dd21609ba499e11bcc2fc777e 2025-03-14T07:49:44.7024472Z deleted: sha256:7f5fe1c39d59479132ac6cdf012d3316fb0cb99134e8f07239b3624825a5faa9 2025-03-14T07:49:44.7025298Z deleted: sha256:c753683add3a6e4b1dda97c0f6a64c8d90f67260f82e4106341a56d637543f0e 2025-03-14T07:49:44.7026047Z deleted: sha256:c4074c1171dd67a89d56a1a44eb56892bff0501ad6f438beca08f599d3725154 2025-03-14T07:49:44.7026799Z deleted: sha256:d4ab5bb476515ad979b863042da19881b668a756385caacbf21d8ac0df2425ac 2025-03-14T07:49:44.7027553Z deleted: sha256:2c2814dde2e80ba2f7122b4ff39392fa0186b31df60c892abec6c9f187bd31bf 2025-03-14T07:49:44.7028315Z deleted: sha256:af3d18449f2d83db98fad6c287d4ae00370ed0fa8670bc4c9eb7b6e054a92eeb 2025-03-14T07:49:44.7029080Z deleted: sha256:e5673c23789df3e5c405fcafa2c47deae3fa691f80ff2c72412422e22e39528e 2025-03-14T07:49:44.7029847Z deleted: sha256:65bcdebae4e462a125146eacdbc728fb2ff87064993e5a54d29df7ede306e968 2025-03-14T07:49:44.7030599Z deleted: sha256:372600f14866b810377df678b82dc565002aafc83280e4ad80f5d5b3dfa17344 2025-03-14T07:49:44.7031347Z deleted: sha256:524046386cfad352a7ac314a15633be3770ad341e2a52fae2b9cbdac84cc724b 2025-03-14T07:49:44.7032101Z deleted: sha256:3f5a5727047ed4d2b37f3d7986c0a814b214c6e1b5012d7256e066cda7b0062d 2025-03-14T07:49:44.7032843Z deleted: sha256:52703cb259a066b08e85d50a3cd88c2c9f34badf43a3c9615d82439995192a25 2025-03-14T07:49:44.7033855Z deleted: sha256:aaa310bd2d0ac6cde807e0075216db34c2e435ff069ddd2e42d0c63ba732c4c1 2025-03-14T07:49:44.7034649Z deleted: sha256:1bea0800445b17eb580cec7bad0f81b425dc9029defe884ac9e344dcdac573ce 2025-03-14T07:49:44.7035416Z deleted: sha256:a0116f0a8a57dee9e2434ef87932a2930dbaa2f45a8da4e215a707135f9f1c5e 2025-03-14T07:49:44.7036189Z deleted: sha256:4e4f830198665bf060eebc820766c03bfdbc66ec605f4e265fe7e32063f9579c 2025-03-14T07:49:44.7036970Z deleted: sha256:f829a78187b6663e8ad2a9f840f875e87ccbdaf9bd5694ce3b8dee4be066f67a 2025-03-14T07:49:44.7037752Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2025-03-14T07:49:44.7038533Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2025-03-14T07:49:44.7039315Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2025-03-14T07:49:44.7040086Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2025-03-14T07:49:44.7040866Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2025-03-14T07:49:44.7041655Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2025-03-14T07:49:44.7042438Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2025-03-14T07:49:44.7043203Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2025-03-14T07:49:44.7043981Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2025-03-14T07:49:44.7044832Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2025-03-14T07:49:44.7045807Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2025-03-14T07:49:44.7046224Z 2025-03-14T07:49:44.7046368Z Total reclaimed space: 43.5GB 2025-03-14T07:49:44.7138360Z Post job cleanup. 2025-03-14T07:49:44.7196095Z Post job cleanup. 2025-03-14T07:49:44.8210736Z [command]/usr/bin/git version 2025-03-14T07:49:44.8251023Z git version 2.47.1 2025-03-14T07:49:44.8289246Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/0665bbf6-de8f-4e72-8022-3c4052ffdcbf/.gitconfig' 2025-03-14T07:49:44.8298925Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/0665bbf6-de8f-4e72-8022-3c4052ffdcbf' before making global git config changes 2025-03-14T07:49:44.8300044Z Adding repository directory to the temporary git global config as a safe directory 2025-03-14T07:49:44.8304384Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-03-14T07:49:44.8344735Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-03-14T07:49:44.8376821Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-03-14T07:49:44.8698497Z Entering 'android/libs/fbjni' 2025-03-14T07:49:44.8754659Z Entering 'third_party/FP16' 2025-03-14T07:49:44.8812373Z Entering 'third_party/FXdiv' 2025-03-14T07:49:44.8868600Z Entering 'third_party/NNPACK' 2025-03-14T07:49:44.8930702Z Entering 'third_party/NVTX' 2025-03-14T07:49:44.8988789Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T07:49:44.9044955Z Entering 'third_party/XNNPACK' 2025-03-14T07:49:44.9119456Z Entering 'third_party/benchmark' 2025-03-14T07:49:44.9174296Z Entering 'third_party/composable_kernel' 2025-03-14T07:49:44.9236553Z Entering 'third_party/cpp-httplib' 2025-03-14T07:49:44.9294062Z Entering 'third_party/cpuinfo' 2025-03-14T07:49:44.9351196Z Entering 'third_party/cudnn_frontend' 2025-03-14T07:49:44.9411968Z Entering 'third_party/cutlass' 2025-03-14T07:49:44.9476889Z Entering 'third_party/eigen' 2025-03-14T07:49:44.9538066Z Entering 'third_party/fbgemm' 2025-03-14T07:49:44.9595145Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T07:49:44.9651643Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T07:49:44.9714807Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T07:49:44.9776773Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T07:49:44.9835262Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T07:49:44.9895075Z Entering 'third_party/flash-attention' 2025-03-14T07:49:44.9952043Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T07:49:45.0023584Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T07:49:45.0093962Z Entering 'third_party/flatbuffers' 2025-03-14T07:49:45.0153615Z Entering 'third_party/fmt' 2025-03-14T07:49:45.0212629Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T07:49:45.0268933Z Entering 'third_party/gloo' 2025-03-14T07:49:45.0328551Z Entering 'third_party/googletest' 2025-03-14T07:49:45.0387007Z Entering 'third_party/ideep' 2025-03-14T07:49:45.0441142Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T07:49:45.0506917Z Entering 'third_party/ittapi' 2025-03-14T07:49:45.0562876Z Entering 'third_party/kineto' 2025-03-14T07:49:45.0626221Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T07:49:45.0684308Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T07:49:45.0746953Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T07:49:45.0803399Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T07:49:45.0860996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T07:49:45.0915268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T07:49:45.0974252Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T07:49:45.1033058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T07:49:45.1097949Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T07:49:45.1157613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T07:49:45.1220379Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T07:49:45.1278520Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T07:49:45.1337118Z Entering 'third_party/kleidiai' 2025-03-14T07:49:45.1394219Z Entering 'third_party/mimalloc' 2025-03-14T07:49:45.1450733Z Entering 'third_party/nlohmann' 2025-03-14T07:49:45.1512476Z Entering 'third_party/onnx' 2025-03-14T07:49:45.1593133Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T07:49:45.1657923Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T07:49:45.1715303Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T07:49:45.1773456Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T07:49:45.1830642Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T07:49:45.1886909Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T07:49:45.1944731Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T07:49:45.2002734Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T07:49:45.2058277Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T07:49:45.2114752Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T07:49:45.2173877Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T07:49:45.2234002Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T07:49:45.2312611Z Entering 'third_party/pocketfft' 2025-03-14T07:49:45.2371159Z Entering 'third_party/protobuf' 2025-03-14T07:49:45.2432019Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T07:49:45.2489314Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T07:49:45.2547946Z Entering 'third_party/psimd' 2025-03-14T07:49:45.2608731Z Entering 'third_party/pthreadpool' 2025-03-14T07:49:45.2667616Z Entering 'third_party/pybind11' 2025-03-14T07:49:45.2729329Z Entering 'third_party/python-peachpy' 2025-03-14T07:49:45.2788387Z Entering 'third_party/sleef' 2025-03-14T07:49:45.2845314Z Entering 'third_party/tensorpipe' 2025-03-14T07:49:45.2904959Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T07:49:45.2961026Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T07:49:45.3019984Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T07:49:45.3076125Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T07:49:45.3132616Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T07:49:45.3209828Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-03-14T07:49:45.3230406Z http.https://github.com/.extraheader 2025-03-14T07:49:45.3239171Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-03-14T07:49:45.3268818Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-03-14T07:49:45.3587126Z Entering 'android/libs/fbjni' 2025-03-14T07:49:45.3627071Z http.https://github.com/.extraheader 2025-03-14T07:49:45.3663783Z Entering 'third_party/FP16' 2025-03-14T07:49:45.3703319Z http.https://github.com/.extraheader 2025-03-14T07:49:45.3740924Z Entering 'third_party/FXdiv' 2025-03-14T07:49:45.3779421Z http.https://github.com/.extraheader 2025-03-14T07:49:45.3814484Z Entering 'third_party/NNPACK' 2025-03-14T07:49:45.3852523Z http.https://github.com/.extraheader 2025-03-14T07:49:45.3888325Z Entering 'third_party/NVTX' 2025-03-14T07:49:45.3925194Z http.https://github.com/.extraheader 2025-03-14T07:49:45.3968866Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T07:49:45.4007842Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4045270Z Entering 'third_party/XNNPACK' 2025-03-14T07:49:45.4084701Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4144559Z Entering 'third_party/benchmark' 2025-03-14T07:49:45.4185583Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4220841Z Entering 'third_party/composable_kernel' 2025-03-14T07:49:45.4260458Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4304718Z Entering 'third_party/cpp-httplib' 2025-03-14T07:49:45.4343410Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4377840Z Entering 'third_party/cpuinfo' 2025-03-14T07:49:45.4417967Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4454172Z Entering 'third_party/cudnn_frontend' 2025-03-14T07:49:45.4494305Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4531060Z Entering 'third_party/cutlass' 2025-03-14T07:49:45.4571101Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4619638Z Entering 'third_party/eigen' 2025-03-14T07:49:45.4658568Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4696581Z Entering 'third_party/fbgemm' 2025-03-14T07:49:45.4735911Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4774396Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T07:49:45.4813131Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4854619Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T07:49:45.4892876Z http.https://github.com/.extraheader 2025-03-14T07:49:45.4935742Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T07:49:45.4971412Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5018749Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T07:49:45.5058013Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5093108Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T07:49:45.5131172Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5176003Z Entering 'third_party/flash-attention' 2025-03-14T07:49:45.5215165Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5253089Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T07:49:45.5292496Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5340053Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T07:49:45.5377521Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5427006Z Entering 'third_party/flatbuffers' 2025-03-14T07:49:45.5465997Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5506453Z Entering 'third_party/fmt' 2025-03-14T07:49:45.5545185Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5579373Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T07:49:45.5619081Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5653992Z Entering 'third_party/gloo' 2025-03-14T07:49:45.5694292Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5736475Z Entering 'third_party/googletest' 2025-03-14T07:49:45.5773102Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5810879Z Entering 'third_party/ideep' 2025-03-14T07:49:45.5849304Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5888115Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T07:49:45.5925148Z http.https://github.com/.extraheader 2025-03-14T07:49:45.5974219Z Entering 'third_party/ittapi' 2025-03-14T07:49:45.6014263Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6050904Z Entering 'third_party/kineto' 2025-03-14T07:49:45.6089567Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6128814Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T07:49:45.6167549Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6209030Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T07:49:45.6244246Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6283277Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T07:49:45.6322704Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6362749Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T07:49:45.6400508Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6435392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T07:49:45.6472784Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6511698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T07:49:45.6549768Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6590200Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T07:49:45.6628551Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6666804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T07:49:45.6705252Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6741658Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T07:49:45.6778805Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6818535Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T07:49:45.6859441Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6897979Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T07:49:45.6935532Z http.https://github.com/.extraheader 2025-03-14T07:49:45.6974507Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T07:49:45.7013415Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7052038Z Entering 'third_party/kleidiai' 2025-03-14T07:49:45.7091646Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7129311Z Entering 'third_party/mimalloc' 2025-03-14T07:49:45.7167707Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7207874Z Entering 'third_party/nlohmann' 2025-03-14T07:49:45.7246257Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7282645Z Entering 'third_party/onnx' 2025-03-14T07:49:45.7320110Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7381218Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T07:49:45.7423301Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7461257Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T07:49:45.7501306Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7545496Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T07:49:45.7584787Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7619205Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T07:49:45.7656397Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7691548Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T07:49:45.7729172Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7768414Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T07:49:45.7806635Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7847745Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T07:49:45.7883785Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7922742Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T07:49:45.7958730Z http.https://github.com/.extraheader 2025-03-14T07:49:45.7994659Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T07:49:45.8031451Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8071972Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T07:49:45.8109952Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8150048Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T07:49:45.8188853Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8235163Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T07:49:45.8273068Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8337856Z Entering 'third_party/pocketfft' 2025-03-14T07:49:45.8374958Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8412279Z Entering 'third_party/protobuf' 2025-03-14T07:49:45.8449591Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8493034Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T07:49:45.8530144Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8565399Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T07:49:45.8604850Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8648552Z Entering 'third_party/psimd' 2025-03-14T07:49:45.8687588Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8722078Z Entering 'third_party/pthreadpool' 2025-03-14T07:49:45.8761517Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8795425Z Entering 'third_party/pybind11' 2025-03-14T07:49:45.8833841Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8869576Z Entering 'third_party/python-peachpy' 2025-03-14T07:49:45.8907159Z http.https://github.com/.extraheader 2025-03-14T07:49:45.8942524Z Entering 'third_party/sleef' 2025-03-14T07:49:45.8984294Z http.https://github.com/.extraheader 2025-03-14T07:49:45.9019602Z Entering 'third_party/tensorpipe' 2025-03-14T07:49:45.9057738Z http.https://github.com/.extraheader 2025-03-14T07:49:45.9095344Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T07:49:45.9132460Z http.https://github.com/.extraheader 2025-03-14T07:49:45.9173833Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T07:49:45.9212638Z http.https://github.com/.extraheader 2025-03-14T07:49:45.9249266Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T07:49:45.9286518Z http.https://github.com/.extraheader 2025-03-14T07:49:45.9333967Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T07:49:45.9371257Z http.https://github.com/.extraheader 2025-03-14T07:49:45.9407705Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T07:49:45.9444830Z http.https://github.com/.extraheader 2025-03-14T07:49:45.9586476Z A job completed hook has been configured by the self-hosted runner administrator 2025-03-14T07:49:45.9613031Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-03-14T07:49:45.9618662Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:49:45.9619130Z ##[endgroup] 2025-03-14T07:49:45.9706233Z [!ALERT!] Swap in detected! [!ALERT!] 2025-03-14T07:49:57.0301398Z [!ALERT!] Swap out detected [!ALERT!] 2025-03-14T07:50:14.4407587Z Cleaning up orphan processes