2023-11-14T04:44:50.4524362Z Requested labels: linux.8xlarge.nvidia.gpu 2023-11-14T04:44:50.4524515Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/112098/merge 2023-11-14T04:44:50.4524720Z Reusable workflow chain: 2023-11-14T04:44:50.4524818Z pytorch/pytorch/.github/workflows/pull.yml@refs/pull/112098/merge (35b9c00c6974fbcfd0eb53491c53c61a6c439831) 2023-11-14T04:44:50.4524898Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/112098/merge (35b9c00c6974fbcfd0eb53491c53c61a6c439831) 2023-11-14T04:44:50.4524991Z Waiting for a runner to pick up this job... 2023-11-14T04:44:50.6714577Z Job is about to start running on the runner: i-0495cce9e02a09ed6 (organization) 2023-11-14T04:44:55.7764856Z Current runner version: '2.311.0' 2023-11-14T04:44:55.7774274Z Runner name: 'i-0495cce9e02a09ed6' 2023-11-14T04:44:55.7775363Z Runner group name: 'Default' 2023-11-14T04:44:55.7776490Z Machine name: 'ip-10-0-33-25' 2023-11-14T04:44:55.7782504Z ##[group]GITHUB_TOKEN Permissions 2023-11-14T04:44:55.7785224Z Actions: write 2023-11-14T04:44:55.7785891Z Checks: write 2023-11-14T04:44:55.7786646Z Contents: write 2023-11-14T04:44:55.7787328Z Deployments: write 2023-11-14T04:44:55.7787969Z Discussions: write 2023-11-14T04:44:55.7788732Z Issues: write 2023-11-14T04:44:55.7789390Z Metadata: read 2023-11-14T04:44:55.7790009Z Packages: write 2023-11-14T04:44:55.7790711Z Pages: write 2023-11-14T04:44:55.7791379Z PullRequests: write 2023-11-14T04:44:55.7792091Z RepositoryProjects: write 2023-11-14T04:44:55.7792955Z SecurityEvents: write 2023-11-14T04:44:55.7793724Z Statuses: write 2023-11-14T04:44:55.7794354Z ##[endgroup] 2023-11-14T04:44:55.7798951Z Secret source: Actions 2023-11-14T04:44:55.7800125Z Prepare workflow directory 2023-11-14T04:44:56.1557415Z Prepare all required actions 2023-11-14T04:44:56.1804197Z Getting action download info 2023-11-14T04:44:56.3495694Z Download action repository 'pytorch/test-infra@main' (SHA:5fbbad20d6fc6779f80c83977e319b918387cbc0) 2023-11-14T04:44:56.7239251Z Download action repository 'pytorch/pytorch@main' (SHA:b3a76ccc12a0a3cb56fd32a8a14feccf0e2cb918) 2023-11-14T04:45:00.3165156Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2023-11-14T04:45:00.6510553Z Getting action download info 2023-11-14T04:45:00.7516975Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2023-11-14T04:45:00.9707913Z Getting action download info 2023-11-14T04:45:01.1684429Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2023-11-14T04:45:01.3368510Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/112098/merge (35b9c00c6974fbcfd0eb53491c53c61a6c439831) 2023-11-14T04:45:01.3372045Z ##[group] Inputs 2023-11-14T04:45:01.3372617Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:45:01.3374748Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2023-11-14T04:45:01.3377540Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:45:01.3378886Z sync-tag: 2023-11-14T04:45:01.3380068Z timeout-minutes: 360 2023-11-14T04:45:01.3380511Z use-gha: 2023-11-14T04:45:01.3380880Z dashboard-tag: 2023-11-14T04:45:01.3381258Z ##[endgroup] 2023-11-14T04:45:01.3382584Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:45:01.4665352Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2023-11-14T04:45:01.4666044Z with: 2023-11-14T04:45:01.4666787Z github-secret: *** 2023-11-14T04:45:01.4667999Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2023-11-14T04:45:01.4669253Z activate-with-label: false 2023-11-14T04:45:01.4669682Z label: with-ssh 2023-11-14T04:45:01.4670065Z remove-existing-keys: true 2023-11-14T04:45:01.4670468Z env: 2023-11-14T04:45:01.4670794Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:45:01.4671217Z ##[endgroup] 2023-11-14T04:45:01.5967508Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2023-11-14T04:45:01.8859409Z Grabbing public ssh keys from https://github.com/clee2000.keys 2023-11-14T04:45:01.9746517Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2023-11-14T04:45:01.9767706Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2023-11-14T04:45:01.9820525Z Login using: ssh ec2-user@ec2-54-166-62-40.compute-1.amazonaws.com 2023-11-14T04:45:01.9821524Z All testing is done inside the container, to start an interactive session run: 2023-11-14T04:45:01.9822551Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2023-11-14T04:45:02.0160517Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2023-11-14T04:45:02.0161258Z with: 2023-11-14T04:45:02.0161599Z submodules: recursive 2023-11-14T04:45:02.0161999Z fetch-depth: 0 2023-11-14T04:45:02.0162356Z env: 2023-11-14T04:45:02.0162668Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:45:02.0163093Z ##[endgroup] 2023-11-14T04:45:02.0469436Z ##[group]Run retry () { 2023-11-14T04:45:02.0469957Z retry () { 2023-11-14T04:45:02.0470574Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2023-11-14T04:45:02.0471300Z } 2023-11-14T04:45:02.0471684Z echo "${GITHUB_WORKSPACE}" 2023-11-14T04:45:02.0472195Z if [ -z "${NO_SUDO}" ]; then 2023-11-14T04:45:02.0472796Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2023-11-14T04:45:02.0473371Z else 2023-11-14T04:45:02.0473788Z  retry rm -rf "${GITHUB_WORKSPACE}" 2023-11-14T04:45:02.0474323Z fi 2023-11-14T04:45:02.0474763Z mkdir "${GITHUB_WORKSPACE}" 2023-11-14T04:45:02.0504634Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:45:02.0505272Z env: 2023-11-14T04:45:02.0505620Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:45:02.0506052Z NO_SUDO: 2023-11-14T04:45:02.0506379Z ##[endgroup] 2023-11-14T04:45:02.0638641Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:45:05.4524364Z ##[group]Run malfet/checkout@silent-checkout 2023-11-14T04:45:05.4524971Z with: 2023-11-14T04:45:05.4525381Z ref: fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:45:05.4525934Z fetch-depth: 0 2023-11-14T04:45:05.4526318Z submodules: recursive 2023-11-14T04:45:05.4526736Z quiet-checkout: true 2023-11-14T04:45:05.4527149Z repository: pytorch/pytorch 2023-11-14T04:45:05.4527764Z token: *** 2023-11-14T04:45:05.4528124Z ssh-strict: true 2023-11-14T04:45:05.4528505Z persist-credentials: true 2023-11-14T04:45:05.4528941Z clean: true 2023-11-14T04:45:05.4529346Z sparse-checkout-cone-mode: true 2023-11-14T04:45:05.4529810Z lfs: false 2023-11-14T04:45:05.4530180Z set-safe-directory: true 2023-11-14T04:45:05.4530591Z env: 2023-11-14T04:45:05.4531184Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:45:05.4531612Z ##[endgroup] 2023-11-14T04:45:05.5857066Z Syncing repository: pytorch/pytorch 2023-11-14T04:45:05.5859245Z ##[group]Getting Git version info 2023-11-14T04:45:05.5860167Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2023-11-14T04:45:05.5861250Z [command]/usr/bin/git version 2023-11-14T04:45:05.5861693Z git version 2.40.1 2023-11-14T04:45:05.5868564Z ##[endgroup] 2023-11-14T04:45:05.5891534Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/6f1f4dd4-050f-43b5-9839-dba8e4e30644' before making global git config changes 2023-11-14T04:45:05.5893164Z Adding repository directory to the temporary git global config as a safe directory 2023-11-14T04:45:05.5898990Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:45:05.5944840Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2023-11-14T04:45:05.5951071Z ##[group]Initializing the repository 2023-11-14T04:45:05.5955037Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:45:05.5987960Z hint: Using 'master' as the name for the initial branch. This default branch name 2023-11-14T04:45:05.5989632Z hint: is subject to change. To configure the initial branch name to use in all 2023-11-14T04:45:05.5990712Z hint: of your new repositories, which will suppress this warning, call: 2023-11-14T04:45:05.5991756Z hint: 2023-11-14T04:45:05.5992567Z hint: git config --global init.defaultBranch 2023-11-14T04:45:05.5993221Z hint: 2023-11-14T04:45:05.5993912Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-11-14T04:45:05.5995039Z hint: 'development'. The just-created branch can be renamed via this command: 2023-11-14T04:45:05.5996004Z hint: 2023-11-14T04:45:05.5996480Z hint: git branch -m 2023-11-14T04:45:05.5997485Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2023-11-14T04:45:05.6001312Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2023-11-14T04:45:05.6035456Z ##[endgroup] 2023-11-14T04:45:05.6036194Z ##[group]Disabling automatic garbage collection 2023-11-14T04:45:05.6039482Z [command]/usr/bin/git config --local gc.auto 0 2023-11-14T04:45:05.6074017Z ##[endgroup] 2023-11-14T04:45:05.6074669Z ##[group]Setting up auth 2023-11-14T04:45:05.6083562Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2023-11-14T04:45:05.6118385Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2023-11-14T04:45:05.6421666Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2023-11-14T04:45:05.6453024Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2023-11-14T04:45:05.6753932Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2023-11-14T04:45:05.6803236Z ##[endgroup] 2023-11-14T04:45:05.6803918Z ##[group]Fetching the repository 2023-11-14T04:45:05.6812820Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2023-11-14T04:45:13.4780275Z remote: Enumerating objects: 1197588 2023-11-14T04:45:13.4781210Z remote: Enumerating objects: 1200035, done. 2023-11-14T04:45:13.4782145Z remote: Counting objects: 0% (1/2447) 2023-11-14T04:45:13.4782786Z remote: Counting objects: 1% (25/2447) 2023-11-14T04:45:13.4783445Z remote: Counting objects: 2% (49/2447) 2023-11-14T04:45:13.4784080Z remote: Counting objects: 3% (74/2447) 2023-11-14T04:45:13.4785015Z remote: Counting objects: 4% (98/2447) 2023-11-14T04:45:13.4785653Z remote: Counting objects: 5% (123/2447) 2023-11-14T04:45:13.4786293Z remote: Counting objects: 6% (147/2447) 2023-11-14T04:45:13.4786896Z remote: Counting objects: 7% (172/2447) 2023-11-14T04:45:13.4787525Z remote: Counting objects: 8% (196/2447) 2023-11-14T04:45:13.4788166Z remote: Counting objects: 9% (221/2447) 2023-11-14T04:45:13.4788816Z remote: Counting objects: 10% (245/2447) 2023-11-14T04:45:13.4789427Z remote: Counting objects: 11% (270/2447) 2023-11-14T04:45:13.4790055Z remote: Counting objects: 12% (294/2447) 2023-11-14T04:45:13.4790677Z remote: Counting objects: 13% (319/2447) 2023-11-14T04:45:13.4791278Z remote: Counting objects: 14% (343/2447) 2023-11-14T04:45:13.4791910Z remote: Counting objects: 15% (368/2447) 2023-11-14T04:45:13.4792531Z remote: Counting objects: 16% (392/2447) 2023-11-14T04:45:13.4793133Z remote: Counting objects: 17% (416/2447) 2023-11-14T04:45:13.4794030Z remote: Counting objects: 18% (441/2447) 2023-11-14T04:45:13.4794660Z remote: Counting objects: 19% (465/2447) 2023-11-14T04:45:13.4795262Z remote: Counting objects: 20% (490/2447) 2023-11-14T04:45:13.4795891Z remote: Counting objects: 21% (514/2447) 2023-11-14T04:45:13.4796509Z remote: Counting objects: 22% (539/2447) 2023-11-14T04:45:13.4797494Z remote: Counting objects: 23% (563/2447) 2023-11-14T04:45:13.4798119Z remote: Counting objects: 24% (588/2447) 2023-11-14T04:45:13.4798742Z remote: Counting objects: 25% (612/2447) 2023-11-14T04:45:13.4799393Z remote: Counting objects: 26% (637/2447) 2023-11-14T04:45:13.4799995Z remote: Counting objects: 27% (661/2447) 2023-11-14T04:45:13.4800772Z remote: Counting objects: 28% (686/2447) 2023-11-14T04:45:13.4801423Z remote: Counting objects: 29% (710/2447) 2023-11-14T04:45:13.4802025Z remote: Counting objects: 30% (735/2447) 2023-11-14T04:45:13.4802651Z remote: Counting objects: 31% (759/2447) 2023-11-14T04:45:13.4803269Z remote: Counting objects: 32% (784/2447) 2023-11-14T04:45:13.4803875Z remote: Counting objects: 33% (808/2447) 2023-11-14T04:45:13.4804496Z remote: Counting objects: 34% (832/2447) 2023-11-14T04:45:13.4805125Z remote: Counting objects: 35% (857/2447) 2023-11-14T04:45:13.4805737Z remote: Counting objects: 36% (881/2447) 2023-11-14T04:45:13.4806362Z remote: Counting objects: 37% (906/2447) 2023-11-14T04:45:13.4806985Z remote: Counting objects: 38% (930/2447) 2023-11-14T04:45:13.4807589Z remote: Counting objects: 39% (955/2447) 2023-11-14T04:45:13.4808210Z remote: Counting objects: 40% (979/2447) 2023-11-14T04:45:13.4808848Z remote: Counting objects: 41% (1004/2447) 2023-11-14T04:45:13.4809505Z remote: Counting objects: 42% (1028/2447) 2023-11-14T04:45:13.4810141Z remote: Counting objects: 43% (1053/2447) 2023-11-14T04:45:13.4811243Z remote: Counting objects: 44% (1077/2447) 2023-11-14T04:45:13.4811915Z remote: Counting objects: 45% (1102/2447) 2023-11-14T04:45:13.4812542Z remote: Counting objects: 46% (1126/2447) 2023-11-14T04:45:13.4813184Z remote: Counting objects: 47% (1151/2447) 2023-11-14T04:45:13.4813933Z remote: Counting objects: 48% (1175/2447) 2023-11-14T04:45:13.4814566Z remote: Counting objects: 49% (1200/2447) 2023-11-14T04:45:13.4815204Z remote: Counting objects: 50% (1224/2447) 2023-11-14T04:45:13.4815847Z remote: Counting objects: 51% (1248/2447) 2023-11-14T04:45:13.4816469Z remote: Counting objects: 52% (1273/2447) 2023-11-14T04:45:13.4817116Z remote: Counting objects: 53% (1297/2447) 2023-11-14T04:45:13.4817763Z remote: Counting objects: 54% (1322/2447) 2023-11-14T04:45:13.4818391Z remote: Counting objects: 55% (1346/2447) 2023-11-14T04:45:13.4819030Z remote: Counting objects: 56% (1371/2447) 2023-11-14T04:45:13.4819670Z remote: Counting objects: 57% (1395/2447) 2023-11-14T04:45:13.4820292Z remote: Counting objects: 58% (1420/2447) 2023-11-14T04:45:13.4820937Z remote: Counting objects: 59% (1444/2447) 2023-11-14T04:45:13.4821582Z remote: Counting objects: 60% (1469/2447) 2023-11-14T04:45:13.4822221Z remote: Counting objects: 61% (1493/2447) 2023-11-14T04:45:13.4822857Z remote: Counting objects: 62% (1518/2447) 2023-11-14T04:45:13.4823511Z remote: Counting objects: 63% (1542/2447) 2023-11-14T04:45:13.4824156Z remote: Counting objects: 64% (1567/2447) 2023-11-14T04:45:13.4824831Z remote: Counting objects: 65% (1591/2447) 2023-11-14T04:45:13.4825465Z remote: Counting objects: 66% (1616/2447) 2023-11-14T04:45:13.4826105Z remote: Counting objects: 67% (1640/2447) 2023-11-14T04:45:13.4826750Z remote: Counting objects: 68% (1664/2447) 2023-11-14T04:45:13.4827374Z remote: Counting objects: 69% (1689/2447) 2023-11-14T04:45:13.4828017Z remote: Counting objects: 70% (1713/2447) 2023-11-14T04:45:13.4828661Z remote: Counting objects: 71% (1738/2447) 2023-11-14T04:45:13.4829283Z remote: Counting objects: 72% (1762/2447) 2023-11-14T04:45:13.4829924Z remote: Counting objects: 73% (1787/2447) 2023-11-14T04:45:13.4830748Z remote: Counting objects: 74% (1811/2447) 2023-11-14T04:45:13.4831390Z remote: Counting objects: 75% (1836/2447) 2023-11-14T04:45:13.4832015Z remote: Counting objects: 76% (1860/2447) 2023-11-14T04:45:13.4832654Z remote: Counting objects: 77% (1885/2447) 2023-11-14T04:45:13.4833290Z remote: Counting objects: 78% (1909/2447) 2023-11-14T04:45:13.4834028Z remote: Counting objects: 79% (1934/2447) 2023-11-14T04:45:13.4834697Z remote: Counting objects: 80% (1958/2447) 2023-11-14T04:45:13.4835336Z remote: Counting objects: 81% (1983/2447) 2023-11-14T04:45:13.4835958Z remote: Counting objects: 82% (2007/2447) 2023-11-14T04:45:13.4836597Z remote: Counting objects: 83% (2032/2447) 2023-11-14T04:45:13.4837237Z remote: Counting objects: 84% (2056/2447) 2023-11-14T04:45:13.4837860Z remote: Counting objects: 85% (2080/2447) 2023-11-14T04:45:13.4838503Z remote: Counting objects: 86% (2105/2447) 2023-11-14T04:45:13.4839151Z remote: Counting objects: 87% (2129/2447) 2023-11-14T04:45:13.4839771Z remote: Counting objects: 88% (2154/2447) 2023-11-14T04:45:13.4840411Z remote: Counting objects: 89% (2178/2447) 2023-11-14T04:45:13.4841047Z remote: Counting objects: 90% (2203/2447) 2023-11-14T04:45:13.4841668Z remote: Counting objects: 91% (2227/2447) 2023-11-14T04:45:13.4842319Z remote: Counting objects: 92% (2252/2447) 2023-11-14T04:45:13.4842958Z remote: Counting objects: 93% (2276/2447) 2023-11-14T04:45:13.4843594Z remote: Counting objects: 94% (2301/2447) 2023-11-14T04:45:13.4844226Z remote: Counting objects: 95% (2325/2447) 2023-11-14T04:45:13.4844863Z remote: Counting objects: 96% (2350/2447) 2023-11-14T04:45:13.4845503Z remote: Counting objects: 97% (2374/2447) 2023-11-14T04:45:13.4846125Z remote: Counting objects: 98% (2399/2447) 2023-11-14T04:45:13.4846760Z remote: Counting objects: 99% (2423/2447) 2023-11-14T04:45:13.4847403Z remote: Counting objects: 100% (2447/2447) 2023-11-14T04:45:13.4848068Z remote: Counting objects: 100% (2447/2447), done. 2023-11-14T04:45:13.4907574Z remote: Compressing objects: 0% (1/1266) 2023-11-14T04:45:13.5321817Z remote: Compressing objects: 1% (13/1266) 2023-11-14T04:45:13.5730383Z remote: Compressing objects: 2% (26/1266) 2023-11-14T04:45:13.7611097Z remote: Compressing objects: 3% (38/1266) 2023-11-14T04:45:13.8794207Z remote: Compressing objects: 4% (51/1266) 2023-11-14T04:45:13.9925929Z remote: Compressing objects: 5% (64/1266) 2023-11-14T04:45:14.0786922Z remote: Compressing objects: 6% (76/1266) 2023-11-14T04:45:14.1515386Z remote: Compressing objects: 7% (89/1266) 2023-11-14T04:45:14.2120829Z remote: Compressing objects: 8% (102/1266) 2023-11-14T04:45:14.2753990Z remote: Compressing objects: 9% (114/1266) 2023-11-14T04:45:14.3178146Z remote: Compressing objects: 10% (127/1266) 2023-11-14T04:45:14.3503167Z remote: Compressing objects: 11% (140/1266) 2023-11-14T04:45:14.3847122Z remote: Compressing objects: 12% (152/1266) 2023-11-14T04:45:14.4048200Z remote: Compressing objects: 13% (165/1266) 2023-11-14T04:45:14.4219269Z remote: Compressing objects: 14% (178/1266) 2023-11-14T04:45:14.4411745Z remote: Compressing objects: 15% (190/1266) 2023-11-14T04:45:14.4590357Z remote: Compressing objects: 16% (203/1266) 2023-11-14T04:45:14.4703496Z remote: Compressing objects: 17% (216/1266) 2023-11-14T04:45:14.4834890Z remote: Compressing objects: 18% (228/1266) 2023-11-14T04:45:14.4846473Z remote: Compressing objects: 18% (240/1266) 2023-11-14T04:45:14.4961550Z remote: Compressing objects: 19% (241/1266) 2023-11-14T04:45:14.5063547Z remote: Compressing objects: 20% (254/1266) 2023-11-14T04:45:14.5136636Z remote: Compressing objects: 21% (266/1266) 2023-11-14T04:45:14.5179918Z remote: Compressing objects: 22% (279/1266) 2023-11-14T04:45:14.5233040Z remote: Compressing objects: 23% (292/1266) 2023-11-14T04:45:14.5277612Z remote: Compressing objects: 24% (304/1266) 2023-11-14T04:45:14.5305481Z remote: Compressing objects: 25% (317/1266) 2023-11-14T04:45:14.5324674Z remote: Compressing objects: 26% (330/1266) 2023-11-14T04:45:14.5331218Z remote: Compressing objects: 27% (342/1266) 2023-11-14T04:45:14.5334229Z remote: Compressing objects: 28% (355/1266) 2023-11-14T04:45:14.5349266Z remote: Compressing objects: 29% (368/1266) 2023-11-14T04:45:14.5357725Z remote: Compressing objects: 30% (380/1266) 2023-11-14T04:45:14.5364687Z remote: Compressing objects: 31% (393/1266) 2023-11-14T04:45:14.5374101Z remote: Compressing objects: 32% (406/1266) 2023-11-14T04:45:14.5380981Z remote: Compressing objects: 33% (418/1266) 2023-11-14T04:45:14.5397023Z remote: Compressing objects: 34% (431/1266) 2023-11-14T04:45:14.5424578Z remote: Compressing objects: 35% (444/1266) 2023-11-14T04:45:14.5439303Z remote: Compressing objects: 36% (456/1266) 2023-11-14T04:45:14.5451569Z remote: Compressing objects: 37% (469/1266) 2023-11-14T04:45:14.5470211Z remote: Compressing objects: 38% (482/1266) 2023-11-14T04:45:14.5480857Z remote: Compressing objects: 39% (494/1266) 2023-11-14T04:45:14.5481543Z remote: Compressing objects: 40% (507/1266) 2023-11-14T04:45:14.5494102Z remote: Compressing objects: 41% (520/1266) 2023-11-14T04:45:14.5506342Z remote: Compressing objects: 42% (532/1266) 2023-11-14T04:45:14.5522710Z remote: Compressing objects: 43% (545/1266) 2023-11-14T04:45:14.5532195Z remote: Compressing objects: 44% (558/1266) 2023-11-14T04:45:14.5535817Z remote: Compressing objects: 45% (570/1266) 2023-11-14T04:45:14.5546929Z remote: Compressing objects: 46% (583/1266) 2023-11-14T04:45:14.5554167Z remote: Compressing objects: 47% (596/1266) 2023-11-14T04:45:14.5562775Z remote: Compressing objects: 48% (608/1266) 2023-11-14T04:45:14.5564951Z remote: Compressing objects: 49% (621/1266) 2023-11-14T04:45:14.5572032Z remote: Compressing objects: 50% (633/1266) 2023-11-14T04:45:14.5578229Z remote: Compressing objects: 51% (646/1266) 2023-11-14T04:45:14.5581589Z remote: Compressing objects: 52% (659/1266) 2023-11-14T04:45:14.5585832Z remote: Compressing objects: 53% (671/1266) 2023-11-14T04:45:14.5588657Z remote: Compressing objects: 54% (684/1266) 2023-11-14T04:45:14.5592102Z remote: Compressing objects: 55% (697/1266) 2023-11-14T04:45:14.5601408Z remote: Compressing objects: 56% (709/1266) 2023-11-14T04:45:14.5607122Z remote: Compressing objects: 57% (722/1266) 2023-11-14T04:45:14.5607803Z remote: Compressing objects: 58% (735/1266) 2023-11-14T04:45:14.5608655Z remote: Compressing objects: 59% (747/1266) 2023-11-14T04:45:14.5609328Z remote: Compressing objects: 60% (760/1266) 2023-11-14T04:45:14.5609999Z remote: Compressing objects: 61% (773/1266) 2023-11-14T04:45:14.5610658Z remote: Compressing objects: 62% (785/1266) 2023-11-14T04:45:14.5611584Z remote: Compressing objects: 63% (798/1266) 2023-11-14T04:45:14.5612258Z remote: Compressing objects: 64% (811/1266) 2023-11-14T04:45:14.5612922Z remote: Compressing objects: 65% (823/1266) 2023-11-14T04:45:14.5613570Z remote: Compressing objects: 66% (836/1266) 2023-11-14T04:45:14.5614229Z remote: Compressing objects: 67% (849/1266) 2023-11-14T04:45:14.5625334Z remote: Compressing objects: 68% (861/1266) 2023-11-14T04:45:14.5637506Z remote: Compressing objects: 69% (874/1266) 2023-11-14T04:45:14.5646920Z remote: Compressing objects: 70% (887/1266) 2023-11-14T04:45:14.5653060Z remote: Compressing objects: 71% (899/1266) 2023-11-14T04:45:14.5660295Z remote: Compressing objects: 72% (912/1266) 2023-11-14T04:45:14.5662440Z remote: Compressing objects: 73% (925/1266) 2023-11-14T04:45:14.5666627Z remote: Compressing objects: 74% (937/1266) 2023-11-14T04:45:14.5672164Z remote: Compressing objects: 75% (950/1266) 2023-11-14T04:45:14.5677164Z remote: Compressing objects: 76% (963/1266) 2023-11-14T04:45:14.5687879Z remote: Compressing objects: 77% (975/1266) 2023-11-14T04:45:14.5688851Z remote: Compressing objects: 78% (988/1266) 2023-11-14T04:45:14.5694960Z remote: Compressing objects: 79% (1001/1266) 2023-11-14T04:45:14.5699352Z remote: Compressing objects: 80% (1013/1266) 2023-11-14T04:45:14.5704371Z remote: Compressing objects: 81% (1026/1266) 2023-11-14T04:45:14.5709224Z remote: Compressing objects: 82% (1039/1266) 2023-11-14T04:45:14.5713411Z remote: Compressing objects: 83% (1051/1266) 2023-11-14T04:45:14.5718389Z remote: Compressing objects: 84% (1064/1266) 2023-11-14T04:45:14.5721817Z remote: Compressing objects: 85% (1077/1266) 2023-11-14T04:45:14.5725426Z remote: Compressing objects: 86% (1089/1266) 2023-11-14T04:45:14.5729056Z remote: Compressing objects: 87% (1102/1266) 2023-11-14T04:45:14.5732346Z remote: Compressing objects: 88% (1115/1266) 2023-11-14T04:45:14.5739431Z remote: Compressing objects: 89% (1127/1266) 2023-11-14T04:45:14.5742662Z remote: Compressing objects: 90% (1140/1266) 2023-11-14T04:45:14.5754664Z remote: Compressing objects: 91% (1153/1266) 2023-11-14T04:45:14.5755356Z remote: Compressing objects: 92% (1165/1266) 2023-11-14T04:45:14.5756026Z remote: Compressing objects: 93% (1178/1266) 2023-11-14T04:45:14.5756681Z remote: Compressing objects: 94% (1191/1266) 2023-11-14T04:45:14.5757395Z remote: Compressing objects: 95% (1203/1266) 2023-11-14T04:45:14.5758070Z remote: Compressing objects: 96% (1216/1266) 2023-11-14T04:45:14.5758748Z remote: Compressing objects: 97% (1229/1266) 2023-11-14T04:45:14.5759420Z remote: Compressing objects: 98% (1241/1266) 2023-11-14T04:45:14.5760095Z remote: Compressing objects: 99% (1254/1266) 2023-11-14T04:45:14.5761459Z remote: Compressing objects: 100% (1266/1266) 2023-11-14T04:45:14.5762176Z remote: Compressing objects: 100% (1266/1266), done. 2023-11-14T04:45:45.5510868Z remote: Total 1200035 (delta 1537), reused 1955 (delta 1171), pack-reused 1197588 2023-11-14T04:46:13.2065481Z [command]/usr/bin/git rev-parse --verify --quiet fe705c0d795479e12d728ea79f708a42b978ffae^{object} 2023-11-14T04:46:13.2093410Z fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:46:13.2099686Z ##[endgroup] 2023-11-14T04:46:13.2100523Z ##[group]Determining the checkout info 2023-11-14T04:46:13.2102090Z ##[endgroup] 2023-11-14T04:46:13.2102841Z ##[group]Checking out the ref 2023-11-14T04:46:13.2107114Z [command]/usr/bin/git checkout --quiet --force fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:46:15.0784207Z ##[endgroup] 2023-11-14T04:46:15.0785741Z ##[group]Setting up auth for fetching submodules 2023-11-14T04:46:15.0791288Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2023-11-14T04:46:15.0848040Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2023-11-14T04:46:15.0881836Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2023-11-14T04:46:15.0916071Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2023-11-14T04:46:15.0946054Z ##[endgroup] 2023-11-14T04:46:15.0946811Z ##[group]Fetching submodules 2023-11-14T04:46:15.0951471Z [command]/usr/bin/git submodule sync --recursive 2023-11-14T04:46:15.1277602Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2023-11-14T04:46:15.1588558Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2023-11-14T04:46:15.1590959Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2023-11-14T04:46:15.1594437Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2023-11-14T04:46:15.1597016Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2023-11-14T04:46:15.1600653Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2023-11-14T04:46:15.1604622Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2023-11-14T04:46:15.1607380Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2023-11-14T04:46:15.1611752Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2023-11-14T04:46:15.1616151Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2023-11-14T04:46:15.1624601Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2023-11-14T04:46:15.1626357Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2023-11-14T04:46:15.1628170Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2023-11-14T04:46:15.1632440Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2023-11-14T04:46:15.1636931Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2023-11-14T04:46:15.1641768Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2023-11-14T04:46:15.1646345Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2023-11-14T04:46:15.1651521Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2023-11-14T04:46:15.1657904Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:46:15.1662744Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2023-11-14T04:46:15.1668156Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2023-11-14T04:46:15.1673444Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2023-11-14T04:46:15.1679099Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2023-11-14T04:46:15.1684722Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2023-11-14T04:46:15.1690478Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2023-11-14T04:46:15.1697259Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2023-11-14T04:46:15.1703115Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2023-11-14T04:46:15.1709401Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2023-11-14T04:46:15.1715598Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2023-11-14T04:46:15.1721966Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2023-11-14T04:46:15.1728643Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2023-11-14T04:46:15.1736223Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2023-11-14T04:46:15.1742934Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2023-11-14T04:46:15.1749907Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2023-11-14T04:46:15.1757076Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2023-11-14T04:46:15.1764080Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2023-11-14T04:46:15.1771859Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2023-11-14T04:46:15.1780206Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2023-11-14T04:46:15.1787735Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2023-11-14T04:46:15.1795599Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2023-11-14T04:46:15.1803379Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2023-11-14T04:46:15.1832416Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2023-11-14T04:46:15.4899285Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2023-11-14T04:46:15.6957939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2023-11-14T04:46:15.8986309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2023-11-14T04:46:16.1772504Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2023-11-14T04:46:16.4637422Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2023-11-14T04:46:18.5319859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2023-11-14T04:46:29.5332586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2023-11-14T04:46:30.0519551Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2023-11-14T04:46:30.7647191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2023-11-14T04:46:32.3789770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2023-11-14T04:46:33.9526400Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2023-11-14T04:46:36.3354890Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2023-11-14T04:46:43.5635726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2023-11-14T04:46:44.7217908Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2023-11-14T04:46:46.7033636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2023-11-14T04:46:48.5571318Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2023-11-14T04:46:48.7524337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2023-11-14T04:46:49.2569963Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2023-11-14T04:46:49.6108222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2023-11-14T04:46:50.7111684Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2023-11-14T04:46:51.0801090Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2023-11-14T04:46:51.3331541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2023-11-14T04:46:51.5896373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2023-11-14T04:46:53.4710422Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2023-11-14T04:46:54.1520998Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2023-11-14T04:46:54.7937570Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2023-11-14T04:46:55.1361382Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2023-11-14T04:47:02.3305252Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2023-11-14T04:47:04.8134593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2023-11-14T04:47:05.3116885Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2023-11-14T04:47:05.5579780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2023-11-14T04:47:14.4507021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2023-11-14T04:47:14.6509775Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2023-11-14T04:47:14.9004843Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2023-11-14T04:47:15.8480755Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2023-11-14T04:47:16.1791521Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2023-11-14T04:47:16.7241366Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2023-11-14T04:47:19.6372244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2023-11-14T04:47:20.1548649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2023-11-14T04:47:23.0477421Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2023-11-14T04:47:23.0602682Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2023-11-14T04:47:23.0699919Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2023-11-14T04:47:23.0984896Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2023-11-14T04:47:23.1258858Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2023-11-14T04:47:23.1755087Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2023-11-14T04:47:24.1348715Z Submodule path 'third_party/XNNPACK': checked out '51a987591a6fc9f0fc0707077f53d763ac132cbf' 2023-11-14T04:47:24.1609540Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2023-11-14T04:47:24.2844173Z Submodule path 'third_party/cpuinfo': checked out '6481e8bef08f606ddd627e4d3be89f64d62e1b8a' 2023-11-14T04:47:24.3278081Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2023-11-14T04:47:24.6867866Z Submodule path 'third_party/cudnn_frontend': checked out '12f35fa2be5994c1106367cac2fba21457b064f4' 2023-11-14T04:47:25.3207058Z Submodule path 'third_party/cutlass': checked out '44c704eae85da352d277d6f092f41412772f70e4' 2023-11-14T04:47:25.6214719Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2023-11-14T04:47:25.6908086Z Submodule path 'third_party/fbgemm': checked out 'd4eec11d72a2280cdf22b77561ba9b6f594cdd7e' 2023-11-14T04:47:25.6926532Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:47:25.6929219Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:47:25.6933701Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:47:25.6936551Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:47:25.6940450Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:47:25.6967922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2023-11-14T04:47:26.6782702Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2023-11-14T04:47:27.3145295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2023-11-14T04:47:29.6781242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2023-11-14T04:47:30.8068709Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2023-11-14T04:47:31.1814701Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2023-11-14T04:47:31.3073896Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2023-11-14T04:47:31.8257297Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2023-11-14T04:47:31.9000456Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2023-11-14T04:47:31.9159625Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2023-11-14T04:47:32.0516046Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2023-11-14T04:47:32.0937451Z Submodule path 'third_party/fmt': checked out 'f5e54359df4c26b6230fc61d38aa294581393084' 2023-11-14T04:47:32.1036935Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2023-11-14T04:47:32.1510173Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2023-11-14T04:47:32.1786603Z Submodule path 'third_party/gloo': checked out 'cf1e1abc95d0b961222ee82b6935f76250fbcf16' 2023-11-14T04:47:32.2343126Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2023-11-14T04:47:32.2480733Z Submodule path 'third_party/ideep': checked out '13dd1fdc6217c9eccfa7166798dd81a0476e63dd' 2023-11-14T04:47:32.2498480Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2023-11-14T04:47:32.2525759Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2023-11-14T04:47:42.6005087Z Submodule path 'third_party/ideep/mkl-dnn': checked out '64f6bcbcbab628e96f33a62c3e975f8535a7bde4' 2023-11-14T04:47:42.6121974Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2023-11-14T04:47:42.6325769Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2023-11-14T04:47:42.7529420Z Submodule path 'third_party/kineto': checked out 'a30ca3f9509c2cfd28561abbca51328f0bdf9014' 2023-11-14T04:47:42.7548769Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:47:42.7551333Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:47:42.7554207Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:47:42.7583540Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2023-11-14T04:47:43.2907752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2023-11-14T04:47:44.5586003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2023-11-14T04:47:45.7668720Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2023-11-14T04:47:45.7688552Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:47:45.7693540Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:47:45.7697458Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:47:45.7701996Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:47:45.7706427Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:47:45.7711200Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:47:45.7715746Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:47:45.7720628Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:47:45.7749963Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2023-11-14T04:47:46.5816160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2023-11-14T04:47:46.9705740Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2023-11-14T04:47:48.2486774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2023-11-14T04:47:48.5806842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2023-11-14T04:47:49.1047152Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2023-11-14T04:47:50.2124471Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2023-11-14T04:47:58.7146819Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2023-11-14T04:47:59.1868314Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2023-11-14T04:47:59.2074486Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2023-11-14T04:47:59.2528829Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2023-11-14T04:47:59.2676228Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2023-11-14T04:47:59.2694275Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:47:59.2721646Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2023-11-14T04:47:59.6302929Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2023-11-14T04:47:59.6513717Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2023-11-14T04:47:59.6995984Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2023-11-14T04:47:59.8298555Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2023-11-14T04:47:59.8483040Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2023-11-14T04:47:59.8890298Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2023-11-14T04:47:59.9563878Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2023-11-14T04:47:59.9983006Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2023-11-14T04:48:00.0246300Z Submodule path 'third_party/nccl/nccl': checked out '8c6c5951854a57ba90c4424fa040497f6defac46' 2023-11-14T04:48:00.0402715Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2023-11-14T04:48:00.1722768Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2023-11-14T04:48:00.5856721Z Submodule path 'third_party/onnx': checked out 'ccde5da81388ffa770ca98b64e07f803ad089414' 2023-11-14T04:48:00.5894134Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:00.5897334Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:00.5925590Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2023-11-14T04:48:01.1355479Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2023-11-14T04:48:02.1213007Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2023-11-14T04:48:02.1619893Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2023-11-14T04:48:02.1821429Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2023-11-14T04:48:02.1838575Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:02.1865551Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2023-11-14T04:48:04.8117963Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2023-11-14T04:48:04.8139898Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:04.8142757Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:04.8170311Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2023-11-14T04:48:05.3340979Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2023-11-14T04:48:06.2866277Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2023-11-14T04:48:06.3710174Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2023-11-14T04:48:06.3727587Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:06.3755091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2023-11-14T04:48:06.6227756Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2023-11-14T04:48:06.6333894Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2023-11-14T04:48:06.9567238Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2023-11-14T04:48:06.9590756Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:06.9593690Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:06.9622714Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2023-11-14T04:48:07.4528692Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2023-11-14T04:48:08.6121632Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2023-11-14T04:48:08.6968996Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2023-11-14T04:48:08.7063937Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2023-11-14T04:48:08.7192824Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2023-11-14T04:48:08.7605649Z Submodule path 'third_party/pybind11': checked out '8a099e44b3d5f85b20f05828d919d2332a8de841' 2023-11-14T04:48:08.7942625Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2023-11-14T04:48:08.8459555Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2023-11-14T04:48:08.9866208Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2023-11-14T04:48:09.0171338Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2023-11-14T04:48:09.0189687Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:09.0192298Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:09.0195767Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:09.0199108Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:09.0228132Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2023-11-14T04:48:10.1227320Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2023-11-14T04:48:10.3845808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2023-11-14T04:48:12.0689898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2023-11-14T04:48:13.0865283Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2023-11-14T04:48:13.1038041Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2023-11-14T04:48:13.1792940Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2023-11-14T04:48:13.2133632Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2023-11-14T04:48:13.2150058Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:13.2178530Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2023-11-14T04:48:13.6798166Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2023-11-14T04:48:13.8538551Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2023-11-14T04:48:13.8572611Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2023-11-14T04:48:13.8902047Z Entering 'android/libs/fbjni' 2023-11-14T04:48:13.8946934Z Entering 'third_party/FP16' 2023-11-14T04:48:13.8990333Z Entering 'third_party/FXdiv' 2023-11-14T04:48:13.9034879Z Entering 'third_party/NNPACK' 2023-11-14T04:48:13.9079492Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:13.9123518Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:13.9167266Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:13.9227112Z Entering 'third_party/benchmark' 2023-11-14T04:48:13.9270595Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:13.9315488Z Entering 'third_party/cub' 2023-11-14T04:48:13.9360014Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:13.9409631Z Entering 'third_party/cutlass' 2023-11-14T04:48:13.9461951Z Entering 'third_party/eigen' 2023-11-14T04:48:13.9509232Z Entering 'third_party/fbgemm' 2023-11-14T04:48:13.9553885Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:13.9597521Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:13.9641182Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:13.9691678Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:13.9735845Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:13.9781005Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:13.9829071Z Entering 'third_party/fmt' 2023-11-14T04:48:13.9872226Z Entering 'third_party/foxi' 2023-11-14T04:48:13.9915289Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:13.9959861Z Entering 'third_party/gloo' 2023-11-14T04:48:14.0004009Z Entering 'third_party/googletest' 2023-11-14T04:48:14.0048365Z Entering 'third_party/ideep' 2023-11-14T04:48:14.0090630Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:14.0139963Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:14.0184370Z Entering 'third_party/ittapi' 2023-11-14T04:48:14.0227706Z Entering 'third_party/kineto' 2023-11-14T04:48:14.0271423Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:14.0315451Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:14.0362297Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:14.0405797Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:14.0450128Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:14.0493998Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:14.0538320Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:14.0581312Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:14.0624500Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:14.0669220Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:14.0713397Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:14.0756563Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:14.0801548Z Entering 'third_party/mimalloc' 2023-11-14T04:48:14.0844833Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:14.0888680Z Entering 'third_party/neon2sse' 2023-11-14T04:48:14.0932519Z Entering 'third_party/nlohmann' 2023-11-14T04:48:14.0979382Z Entering 'third_party/onnx' 2023-11-14T04:48:14.1040843Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:14.1084726Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:14.1131969Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:14.1175894Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:14.1225332Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:14.1269404Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:14.1312748Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:14.1360845Z Entering 'third_party/pocketfft' 2023-11-14T04:48:14.1404613Z Entering 'third_party/protobuf' 2023-11-14T04:48:14.1452942Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:14.1495508Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:14.1540038Z Entering 'third_party/psimd' 2023-11-14T04:48:14.1584327Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:14.1627007Z Entering 'third_party/pybind11' 2023-11-14T04:48:14.1669990Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:14.1714366Z Entering 'third_party/sleef' 2023-11-14T04:48:14.1758167Z Entering 'third_party/tbb' 2023-11-14T04:48:14.1803358Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:14.1846872Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:14.1889346Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:14.1930735Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:14.1973781Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:14.2015513Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:14.2061558Z Entering 'third_party/zstd' 2023-11-14T04:48:14.2120982Z ##[endgroup] 2023-11-14T04:48:14.2121812Z ##[group]Persisting credentials for submodules 2023-11-14T04:48:14.2126639Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2023-11-14T04:48:14.2445626Z Entering 'android/libs/fbjni' 2023-11-14T04:48:14.2501555Z Entering 'third_party/FP16' 2023-11-14T04:48:14.2556854Z Entering 'third_party/FXdiv' 2023-11-14T04:48:14.2612406Z Entering 'third_party/NNPACK' 2023-11-14T04:48:14.2667664Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:14.2723401Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:14.2779010Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:14.2848031Z Entering 'third_party/benchmark' 2023-11-14T04:48:14.2903738Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:14.2959867Z Entering 'third_party/cub' 2023-11-14T04:48:14.3015583Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:14.3075650Z Entering 'third_party/cutlass' 2023-11-14T04:48:14.3139238Z Entering 'third_party/eigen' 2023-11-14T04:48:14.3197401Z Entering 'third_party/fbgemm' 2023-11-14T04:48:14.3253490Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:14.3308410Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:14.3362534Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:14.3424638Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:14.3478398Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:14.3534605Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:14.3591727Z Entering 'third_party/fmt' 2023-11-14T04:48:14.3647079Z Entering 'third_party/foxi' 2023-11-14T04:48:14.3702200Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:14.3757015Z Entering 'third_party/gloo' 2023-11-14T04:48:14.3813273Z Entering 'third_party/googletest' 2023-11-14T04:48:14.3867108Z Entering 'third_party/ideep' 2023-11-14T04:48:14.3921035Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:14.3982096Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:14.4036704Z Entering 'third_party/ittapi' 2023-11-14T04:48:14.4091072Z Entering 'third_party/kineto' 2023-11-14T04:48:14.4146312Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:14.4202197Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:14.4259837Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:14.4315774Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:14.4370178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:14.4425661Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:14.4483215Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:14.4538744Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:14.4594318Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:14.4651396Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:14.4707725Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:14.4763423Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:14.4820085Z Entering 'third_party/mimalloc' 2023-11-14T04:48:14.4875264Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:14.4930455Z Entering 'third_party/neon2sse' 2023-11-14T04:48:14.4984957Z Entering 'third_party/nlohmann' 2023-11-14T04:48:14.5040669Z Entering 'third_party/onnx' 2023-11-14T04:48:14.5114434Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:14.5170047Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:14.5228467Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:14.5283763Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:14.5344168Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:14.5399151Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:14.5454282Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:14.5514517Z Entering 'third_party/pocketfft' 2023-11-14T04:48:14.5569041Z Entering 'third_party/protobuf' 2023-11-14T04:48:14.5628255Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:14.5682567Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:14.5739665Z Entering 'third_party/psimd' 2023-11-14T04:48:14.5794560Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:14.5848781Z Entering 'third_party/pybind11' 2023-11-14T04:48:14.5904245Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:14.5959766Z Entering 'third_party/sleef' 2023-11-14T04:48:14.6015788Z Entering 'third_party/tbb' 2023-11-14T04:48:14.6074159Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:14.6130333Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:14.6185774Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:14.6241562Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:14.6297714Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:14.6352814Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:14.6410380Z Entering 'third_party/zstd' 2023-11-14T04:48:14.6481043Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2023-11-14T04:48:14.6797120Z Entering 'android/libs/fbjni' 2023-11-14T04:48:14.6849007Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2023-11-14T04:48:14.6867616Z Entering 'third_party/FP16' 2023-11-14T04:48:14.6920000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2023-11-14T04:48:14.6941063Z Entering 'third_party/FXdiv' 2023-11-14T04:48:14.6992802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2023-11-14T04:48:14.7012723Z Entering 'third_party/NNPACK' 2023-11-14T04:48:14.7064409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2023-11-14T04:48:14.7083880Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:14.7135714Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2023-11-14T04:48:14.7154498Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:14.7206090Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2023-11-14T04:48:14.7226139Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:14.7277908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2023-11-14T04:48:14.7311505Z Entering 'third_party/benchmark' 2023-11-14T04:48:14.7362827Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2023-11-14T04:48:14.7382504Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:14.7434093Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2023-11-14T04:48:14.7454077Z Entering 'third_party/cub' 2023-11-14T04:48:14.7505075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2023-11-14T04:48:14.7523495Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:14.7575083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2023-11-14T04:48:14.7599635Z Entering 'third_party/cutlass' 2023-11-14T04:48:14.7651144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2023-11-14T04:48:14.7677466Z Entering 'third_party/eigen' 2023-11-14T04:48:14.7729565Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2023-11-14T04:48:14.7751097Z Entering 'third_party/fbgemm' 2023-11-14T04:48:14.7803322Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2023-11-14T04:48:14.7822428Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:14.7875418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2023-11-14T04:48:14.7893744Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:14.7944386Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2023-11-14T04:48:14.7962994Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:14.8015236Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2023-11-14T04:48:14.8040382Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:14.8092522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:14.8110743Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:14.8162412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2023-11-14T04:48:14.8182090Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:14.8234707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2023-11-14T04:48:14.8256458Z Entering 'third_party/fmt' 2023-11-14T04:48:14.8307196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2023-11-14T04:48:14.8326083Z Entering 'third_party/foxi' 2023-11-14T04:48:14.8377897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2023-11-14T04:48:14.8396418Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:14.8449341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2023-11-14T04:48:14.8468603Z Entering 'third_party/gloo' 2023-11-14T04:48:14.8521076Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2023-11-14T04:48:14.8540249Z Entering 'third_party/googletest' 2023-11-14T04:48:14.8592374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:14.8611646Z Entering 'third_party/ideep' 2023-11-14T04:48:14.8663035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2023-11-14T04:48:14.8681146Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:14.8733244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2023-11-14T04:48:14.8758652Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:14.8809420Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2023-11-14T04:48:14.8828355Z Entering 'third_party/ittapi' 2023-11-14T04:48:14.8880540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2023-11-14T04:48:14.8900024Z Entering 'third_party/kineto' 2023-11-14T04:48:14.8953242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2023-11-14T04:48:14.8972254Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:14.9024872Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2023-11-14T04:48:14.9043242Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:14.9095884Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2023-11-14T04:48:14.9115511Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:14.9167406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2023-11-14T04:48:14.9186453Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:14.9238669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2023-11-14T04:48:14.9258217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:14.9310416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2023-11-14T04:48:14.9328226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:14.9381701Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2023-11-14T04:48:14.9402003Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:14.9453846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2023-11-14T04:48:14.9472111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:14.9525179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:14.9544369Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:14.9596016Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2023-11-14T04:48:14.9615922Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:14.9668066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2023-11-14T04:48:14.9688399Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:14.9740522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2023-11-14T04:48:14.9759089Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:14.9810277Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2023-11-14T04:48:14.9830747Z Entering 'third_party/mimalloc' 2023-11-14T04:48:14.9882408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2023-11-14T04:48:14.9902860Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:14.9954310Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2023-11-14T04:48:14.9973502Z Entering 'third_party/neon2sse' 2023-11-14T04:48:15.0024533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2023-11-14T04:48:15.0042944Z Entering 'third_party/nlohmann' 2023-11-14T04:48:15.0095305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2023-11-14T04:48:15.0115275Z Entering 'third_party/onnx' 2023-11-14T04:48:15.0166830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2023-11-14T04:48:15.0203419Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:15.0258712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2023-11-14T04:48:15.0277334Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:15.0328946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2023-11-14T04:48:15.0350689Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:15.0403451Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2023-11-14T04:48:15.0421889Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:15.0473575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2023-11-14T04:48:15.0497684Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:15.0551213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2023-11-14T04:48:15.0569993Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:15.0622208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2023-11-14T04:48:15.0639947Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:15.0693017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2023-11-14T04:48:15.0716329Z Entering 'third_party/pocketfft' 2023-11-14T04:48:15.0770110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2023-11-14T04:48:15.0789242Z Entering 'third_party/protobuf' 2023-11-14T04:48:15.0840326Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2023-11-14T04:48:15.0863024Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:15.0914706Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2023-11-14T04:48:15.0933599Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:15.0984869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:15.1004993Z Entering 'third_party/psimd' 2023-11-14T04:48:15.1057078Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2023-11-14T04:48:15.1075887Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:15.1127747Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2023-11-14T04:48:15.1146388Z Entering 'third_party/pybind11' 2023-11-14T04:48:15.1197873Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2023-11-14T04:48:15.1218017Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:15.1269513Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2023-11-14T04:48:15.1288661Z Entering 'third_party/sleef' 2023-11-14T04:48:15.1340638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2023-11-14T04:48:15.1360092Z Entering 'third_party/tbb' 2023-11-14T04:48:15.1411495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2023-11-14T04:48:15.1432784Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:15.1485050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2023-11-14T04:48:15.1504083Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:15.1556493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:15.1575269Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:15.1626480Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2023-11-14T04:48:15.1644863Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:15.1697563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2023-11-14T04:48:15.1716229Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:15.1767351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2023-11-14T04:48:15.1785607Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:15.1837797Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2023-11-14T04:48:15.1859913Z Entering 'third_party/zstd' 2023-11-14T04:48:15.1911305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2023-11-14T04:48:15.2936221Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2023-11-14T04:48:15.3254944Z Entering 'android/libs/fbjni' 2023-11-14T04:48:15.3298706Z Entering 'third_party/FP16' 2023-11-14T04:48:15.3341865Z Entering 'third_party/FXdiv' 2023-11-14T04:48:15.3385853Z Entering 'third_party/NNPACK' 2023-11-14T04:48:15.3430095Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:15.3474831Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:15.3519461Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:15.3577059Z Entering 'third_party/benchmark' 2023-11-14T04:48:15.3622165Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:15.3666086Z Entering 'third_party/cub' 2023-11-14T04:48:15.3709604Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:15.3759366Z Entering 'third_party/cutlass' 2023-11-14T04:48:15.3811491Z Entering 'third_party/eigen' 2023-11-14T04:48:15.3857525Z Entering 'third_party/fbgemm' 2023-11-14T04:48:15.3902187Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:15.3945785Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:15.3991870Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:15.4041464Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:15.4084958Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:15.4128604Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:15.4174958Z Entering 'third_party/fmt' 2023-11-14T04:48:15.4220656Z Entering 'third_party/foxi' 2023-11-14T04:48:15.4265342Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:15.4308191Z Entering 'third_party/gloo' 2023-11-14T04:48:15.4352412Z Entering 'third_party/googletest' 2023-11-14T04:48:15.4396834Z Entering 'third_party/ideep' 2023-11-14T04:48:15.4440112Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:15.4490034Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:15.4535520Z Entering 'third_party/ittapi' 2023-11-14T04:48:15.4579376Z Entering 'third_party/kineto' 2023-11-14T04:48:15.4623703Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:15.4668392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:15.4713330Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:15.4757688Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:15.4802714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:15.4845710Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:15.4891638Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:15.4934540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:15.4980182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:15.5025762Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:15.5071003Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:15.5114431Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:15.5159231Z Entering 'third_party/mimalloc' 2023-11-14T04:48:15.5204817Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:15.5248374Z Entering 'third_party/neon2sse' 2023-11-14T04:48:15.5291810Z Entering 'third_party/nlohmann' 2023-11-14T04:48:15.5337987Z Entering 'third_party/onnx' 2023-11-14T04:48:15.5398971Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:15.5442764Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:15.5489654Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:15.5532902Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:15.5581496Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:15.5625525Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:15.5668594Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:15.5717325Z Entering 'third_party/pocketfft' 2023-11-14T04:48:15.5761400Z Entering 'third_party/protobuf' 2023-11-14T04:48:15.5809614Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:15.5853404Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:15.5898976Z Entering 'third_party/psimd' 2023-11-14T04:48:15.5943922Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:15.5987482Z Entering 'third_party/pybind11' 2023-11-14T04:48:15.6030903Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:15.6075010Z Entering 'third_party/sleef' 2023-11-14T04:48:15.6120010Z Entering 'third_party/tbb' 2023-11-14T04:48:15.6165592Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:15.6209559Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:15.6254306Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:15.6297370Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:15.6340534Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:15.6382347Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:15.6428592Z Entering 'third_party/zstd' 2023-11-14T04:48:15.6486452Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2023-11-14T04:48:15.6801022Z Entering 'android/libs/fbjni' 2023-11-14T04:48:15.6843681Z Entering 'third_party/FP16' 2023-11-14T04:48:15.6887018Z Entering 'third_party/FXdiv' 2023-11-14T04:48:15.6930356Z Entering 'third_party/NNPACK' 2023-11-14T04:48:15.6976077Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:15.7020076Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:15.7064031Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:15.7119953Z Entering 'third_party/benchmark' 2023-11-14T04:48:15.7163973Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:15.7209005Z Entering 'third_party/cub' 2023-11-14T04:48:15.7253805Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:15.7302764Z Entering 'third_party/cutlass' 2023-11-14T04:48:15.7353520Z Entering 'third_party/eigen' 2023-11-14T04:48:15.7401124Z Entering 'third_party/fbgemm' 2023-11-14T04:48:15.7445978Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:15.7488418Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:15.7531669Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:15.7582888Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:15.7625078Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:15.7669956Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:15.7717121Z Entering 'third_party/fmt' 2023-11-14T04:48:15.7760708Z Entering 'third_party/foxi' 2023-11-14T04:48:15.7804008Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:15.7847837Z Entering 'third_party/gloo' 2023-11-14T04:48:15.7892366Z Entering 'third_party/googletest' 2023-11-14T04:48:15.7937116Z Entering 'third_party/ideep' 2023-11-14T04:48:15.7979798Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:15.8028453Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:15.8071720Z Entering 'third_party/ittapi' 2023-11-14T04:48:15.8115063Z Entering 'third_party/kineto' 2023-11-14T04:48:15.8158712Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:15.8202682Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:15.8246893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:15.8289638Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:15.8333210Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:15.8376098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:15.8422177Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:15.8464983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:15.8508466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:15.8552104Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:15.8597381Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:15.8640034Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:15.8684748Z Entering 'third_party/mimalloc' 2023-11-14T04:48:15.8728485Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:15.8771923Z Entering 'third_party/neon2sse' 2023-11-14T04:48:15.8816095Z Entering 'third_party/nlohmann' 2023-11-14T04:48:15.8860585Z Entering 'third_party/onnx' 2023-11-14T04:48:15.8921557Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:15.8971888Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:15.9019225Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:15.9063195Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:15.9110875Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:15.9154250Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:15.9197009Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:15.9245165Z Entering 'third_party/pocketfft' 2023-11-14T04:48:15.9289243Z Entering 'third_party/protobuf' 2023-11-14T04:48:15.9337919Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:15.9381411Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:15.9426843Z Entering 'third_party/psimd' 2023-11-14T04:48:15.9471097Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:15.9514389Z Entering 'third_party/pybind11' 2023-11-14T04:48:15.9558926Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:15.9602358Z Entering 'third_party/sleef' 2023-11-14T04:48:15.9646535Z Entering 'third_party/tbb' 2023-11-14T04:48:15.9693058Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:15.9736968Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:15.9779977Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:15.9821960Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:15.9864966Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:15.9907412Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:15.9952862Z Entering 'third_party/zstd' 2023-11-14T04:48:16.0009387Z ##[endgroup] 2023-11-14T04:48:16.0050158Z [command]/usr/bin/git log -1 --format='%H' 2023-11-14T04:48:16.0077641Z 'fe705c0d795479e12d728ea79f708a42b978ffae' 2023-11-14T04:48:16.0282532Z Prepare all required actions 2023-11-14T04:48:16.0283128Z Getting action download info 2023-11-14T04:48:16.1849799Z ##[group]Run ./.github/actions/setup-linux 2023-11-14T04:48:16.1850345Z env: 2023-11-14T04:48:16.1851056Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:16.1851517Z ##[endgroup] 2023-11-14T04:48:16.1940632Z ##[group]Run set -euo pipefail 2023-11-14T04:48:16.1941164Z set -euo pipefail 2023-11-14T04:48:16.1941658Z function get_ec2_metadata() { 2023-11-14T04:48:16.1942313Z  # Pulled from instance metadata endpoint for EC2 2023-11-14T04:48:16.1943449Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2023-11-14T04:48:16.1944441Z  category=$1 2023-11-14T04:48:16.1945110Z  # If it is GCP runner (runner name contains gcp), do not run this 2023-11-14T04:48:16.1945913Z  runner_name_str=i-0495cce9e02a09ed6 2023-11-14T04:48:16.1946632Z  if [[ $runner_name_str != *"gcp"* ]]; then 2023-11-14T04:48:16.1947445Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2023-11-14T04:48:16.1948177Z  else 2023-11-14T04:48:16.1948830Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2023-11-14T04:48:16.1949592Z  fi 2023-11-14T04:48:16.1949927Z } 2023-11-14T04:48:16.1950377Z echo "ami-id: $(get_ec2_metadata ami-id)" 2023-11-14T04:48:16.1951112Z echo "instance-id: $(get_ec2_metadata instance-id)" 2023-11-14T04:48:16.1951920Z echo "instance-type: $(get_ec2_metadata instance-type)" 2023-11-14T04:48:16.1952633Z echo "system info $(uname -a)" 2023-11-14T04:48:16.1961653Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:16.1962296Z env: 2023-11-14T04:48:16.1962618Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:16.1963050Z ##[endgroup] 2023-11-14T04:48:16.2065841Z ami-id: ami-096198a0bccc6bad4 2023-11-14T04:48:16.2131440Z instance-id: i-0495cce9e02a09ed6 2023-11-14T04:48:16.2195529Z instance-type: g3.8xlarge 2023-11-14T04:48:16.2205324Z system info Linux ip-10-0-33-25.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 2023-11-14T04:48:16.2233736Z ##[group]Run if systemctl is-active --quiet docker; then 2023-11-14T04:48:16.2234481Z if systemctl is-active --quiet docker; then 2023-11-14T04:48:16.2235152Z  echo "Docker daemon is running..."; 2023-11-14T04:48:16.2235686Z else 2023-11-14T04:48:16.2236295Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2023-11-14T04:48:16.2237020Z fi 2023-11-14T04:48:16.2244948Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:16.2245559Z env: 2023-11-14T04:48:16.2245897Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:16.2246325Z ##[endgroup] 2023-11-14T04:48:16.2296658Z Docker daemon is running... 2023-11-14T04:48:16.2358225Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T04:48:16.2358985Z with: 2023-11-14T04:48:16.2359422Z shell: bash 2023-11-14T04:48:16.2359876Z timeout_minutes: 5 2023-11-14T04:48:16.2360305Z max_attempts: 3 2023-11-14T04:48:16.2390831Z retry_wait_seconds: 30 2023-11-14T04:48:16.2392637Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2023-11-14T04:48:16.2394490Z polling_interval_seconds: 1 2023-11-14T04:48:16.2394949Z warning_on_retry: true 2023-11-14T04:48:16.2395360Z continue_on_error: false 2023-11-14T04:48:16.2395769Z env: 2023-11-14T04:48:16.2396091Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:16.2396732Z AWS_RETRY_MODE: standard 2023-11-14T04:48:16.2397158Z AWS_MAX_ATTEMPTS: 5 2023-11-14T04:48:16.2397576Z AWS_DEFAULT_REGION: us-east-1 2023-11-14T04:48:16.2398147Z ##[endgroup] 2023-11-14T04:48:17.2472420Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2023-11-14T04:48:17.2474422Z Configure a credential helper to remove this warning. See 2023-11-14T04:48:17.2475781Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2023-11-14T04:48:17.2476456Z 2023-11-14T04:48:17.2476606Z Login Succeeded 2023-11-14T04:48:17.3109179Z Command completed after 1 attempt(s). 2023-11-14T04:48:17.3168718Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-11-14T04:48:17.3169625Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-11-14T04:48:17.3170446Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-11-14T04:48:17.3180034Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:17.3180673Z env: 2023-11-14T04:48:17.3180988Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:17.3181417Z ##[endgroup] 2023-11-14T04:48:17.3261954Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2023-11-14T04:48:17.3262917Z # ignore expansion of "docker ps -q" since it could be empty 2023-11-14T04:48:17.3263652Z # shellcheck disable=SC2046 2023-11-14T04:48:17.3264224Z docker stop $(docker ps -q) || true 2023-11-14T04:48:17.3264801Z # Prune all of the docker images 2023-11-14T04:48:17.3265359Z docker system prune -af 2023-11-14T04:48:17.3273710Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:17.3274330Z env: 2023-11-14T04:48:17.3274646Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:17.3275072Z ##[endgroup] 2023-11-14T04:48:17.3704513Z "docker stop" requires at least 1 argument. 2023-11-14T04:48:17.3705245Z See 'docker stop --help'. 2023-11-14T04:48:17.3705586Z 2023-11-14T04:48:17.3705863Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2023-11-14T04:48:17.3706346Z 2023-11-14T04:48:17.3706533Z Stop one or more running containers 2023-11-14T04:48:17.3941965Z Total reclaimed space: 0B 2023-11-14T04:48:17.3980461Z ##[group]Run set +e 2023-11-14T04:48:17.3980892Z set +e 2023-11-14T04:48:17.3981260Z set -x 2023-11-14T04:48:17.3981614Z  2023-11-14T04:48:17.3981985Z PT_DOMAIN=download.pytorch.org 2023-11-14T04:48:17.3982968Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2023-11-14T04:48:17.3984328Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2023-11-14T04:48:17.3985272Z # one is returned at random 2023-11-14T04:48:17.3985923Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2023-11-14T04:48:17.3986560Z  2023-11-14T04:48:17.3986945Z if [ -z "${RESOLVED_IP}" ]; then 2023-11-14T04:48:17.3987715Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2023-11-14T04:48:17.3988667Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2023-11-14T04:48:17.3989364Z  2023-11-14T04:48:17.3989737Z  if [ -z "${RESOLVED_IP}" ]; then 2023-11-14T04:48:17.3990418Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2023-11-14T04:48:17.3991048Z  exit 1 2023-11-14T04:48:17.3991409Z  fi 2023-11-14T04:48:17.3991752Z fi 2023-11-14T04:48:17.3992082Z  2023-11-14T04:48:17.3992496Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2023-11-14T04:48:17.3993135Z  # Clean up any old records first 2023-11-14T04:48:17.3993761Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2023-11-14T04:48:17.3994297Z fi 2023-11-14T04:48:17.3994629Z  2023-11-14T04:48:17.3995153Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2023-11-14T04:48:17.3996022Z cat /etc/hosts 2023-11-14T04:48:17.4004025Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:17.4004804Z env: 2023-11-14T04:48:17.4005161Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:17.4005569Z ##[endgroup] 2023-11-14T04:48:17.4032738Z + PT_DOMAIN=download.pytorch.org 2023-11-14T04:48:17.4038113Z ++ dig -4 +short download.pytorch.org 2023-11-14T04:48:17.4039014Z ++ tail -n1 2023-11-14T04:48:17.4220153Z + RESOLVED_IP=18.160.10.28 2023-11-14T04:48:17.4220716Z + '[' -z 18.160.10.28 ']' 2023-11-14T04:48:17.4221258Z + grep -r download.pytorch.org /etc/hosts 2023-11-14T04:48:17.4230194Z 18.160.10.36 download.pytorch.org 2023-11-14T04:48:17.4231380Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2023-11-14T04:48:17.4374779Z + echo '18.160.10.28 download.pytorch.org' 2023-11-14T04:48:17.4375359Z + sudo tee -a /etc/hosts 2023-11-14T04:48:17.4629362Z 18.160.10.28 download.pytorch.org 2023-11-14T04:48:17.4645313Z + cat /etc/hosts 2023-11-14T04:48:17.4653055Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2023-11-14T04:48:17.4668476Z ::1 localhost6 localhost6.localdomain6 2023-11-14T04:48:17.4669118Z 18.160.10.28 download.pytorch.org 2023-11-14T04:48:17.4841347Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2023-11-14T04:48:17.4842119Z with: 2023-11-14T04:48:17.4843355Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.4844762Z docker-build-dir: .ci/docker 2023-11-14T04:48:17.4845230Z working-directory: . 2023-11-14T04:48:17.4845796Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2023-11-14T04:48:17.4846432Z env: 2023-11-14T04:48:17.4846766Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:17.4847177Z ##[endgroup] 2023-11-14T04:48:17.4874853Z ##[group]Run set -ex 2023-11-14T04:48:17.4875313Z set -ex 2023-11-14T04:48:17.4875673Z  2023-11-14T04:48:17.4876369Z # If the docker build directory or the build script doesn't exist, the action will 2023-11-14T04:48:17.4877685Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2023-11-14T04:48:17.4878686Z # job could then download the pre-built image as usual 2023-11-14T04:48:17.4879611Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2023-11-14T04:48:17.4880452Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:17.4881225Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:17.4881927Z  2023-11-14T04:48:17.4882570Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2023-11-14T04:48:17.4883354Z  exit 0 2023-11-14T04:48:17.4883709Z else 2023-11-14T04:48:17.4884159Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:17.4884729Z fi 2023-11-14T04:48:17.4885049Z  2023-11-14T04:48:17.4885628Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2023-11-14T04:48:17.4886701Z  # The docker image name already includes the ECR prefix and tag, so we can just 2023-11-14T04:48:17.4887661Z  # use it as it is, but first let's extract the tag 2023-11-14T04:48:17.4888539Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2023-11-14T04:48:17.4889449Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:17.4890305Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:17.4891502Z else 2023-11-14T04:48:17.4892055Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2023-11-14T04:48:17.4892884Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:17.4893988Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:17.4895132Z fi 2023-11-14T04:48:17.4903346Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:17.4903971Z env: 2023-11-14T04:48:17.4904305Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:17.4904729Z REPO_NAME: pytorch 2023-11-14T04:48:17.4906003Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.4907377Z DOCKER_BUILD_DIR: .ci/docker 2023-11-14T04:48:17.4907993Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2023-11-14T04:48:17.4908621Z ##[endgroup] 2023-11-14T04:48:17.4935815Z + [[ ! -d .ci/docker ]] 2023-11-14T04:48:17.4936317Z + [[ ! -f .ci/docker/build.sh ]] 2023-11-14T04:48:17.4936788Z + echo skip=false 2023-11-14T04:48:17.4938899Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2023-11-14T04:48:17.4943994Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.4945409Z ++ awk -F '[:,]' '{print $2}' 2023-11-14T04:48:17.4954141Z + DOCKER_TAG=3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.4955006Z + echo docker-tag=3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.4956923Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.5018856Z ##[group]Run set +e 2023-11-14T04:48:17.5019317Z set +e 2023-11-14T04:48:17.5019680Z set -x 2023-11-14T04:48:17.5020010Z  2023-11-14T04:48:17.5020596Z # Check if image already exists, if it does then skip building it 2023-11-14T04:48:17.5021496Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2023-11-14T04:48:17.5022103Z  exit 0 2023-11-14T04:48:17.5022468Z fi 2023-11-14T04:48:17.5022808Z  2023-11-14T04:48:17.5023411Z # NB: This part requires a full checkout. Otherwise, the merge base will 2023-11-14T04:48:17.5024450Z # be empty. The default action would be to continue rebuild the image 2023-11-14T04:48:17.5025368Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2023-11-14T04:48:17.5026185Z  # if we're on the base branch then use the parent commit 2023-11-14T04:48:17.5026925Z  MERGE_BASE=$(git rev-parse HEAD~) 2023-11-14T04:48:17.5027454Z else 2023-11-14T04:48:17.5028011Z  # otherwise we're on a PR, so use the most recent base commit 2023-11-14T04:48:17.5028866Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2023-11-14T04:48:17.5029498Z fi 2023-11-14T04:48:17.5029832Z  2023-11-14T04:48:17.5030198Z if [[ -z "${MERGE_BASE}" ]]; then 2023-11-14T04:48:17.5030824Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:17.5031382Z  2023-11-14T04:48:17.5032184Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2023-11-14T04:48:17.5033139Z  exit 0 2023-11-14T04:48:17.5033503Z fi 2023-11-14T04:48:17.5033818Z  2023-11-14T04:48:17.5034355Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2023-11-14T04:48:17.5035570Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2023-11-14T04:48:17.5036562Z  exit 1 2023-11-14T04:48:17.5036926Z fi 2023-11-14T04:48:17.5037282Z  2023-11-14T04:48:17.5037885Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2023-11-14T04:48:17.5039231Z # If no image exists but the hash is the same as the previous hash then we should error out here 2023-11-14T04:48:17.5040296Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2023-11-14T04:48:17.5041512Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2023-11-14T04:48:17.5042901Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2023-11-14T04:48:17.5043713Z fi 2023-11-14T04:48:17.5044032Z  2023-11-14T04:48:17.5044465Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:17.5052846Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:17.5053469Z env: 2023-11-14T04:48:17.5053788Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:17.5054235Z DOCKER_BUILD_DIR: .ci/docker 2023-11-14T04:48:17.5054807Z BASE_REVISION: e1c872e009e8aa07281f38723353dc43baf2fc12 2023-11-14T04:48:17.5056242Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.5057829Z DOCKER_TAG: 3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.5058449Z ##[endgroup] 2023-11-14T04:48:17.5085796Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.7271477Z { 2023-11-14T04:48:17.7272176Z "schemaVersion": 2, 2023-11-14T04:48:17.7272874Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2023-11-14T04:48:17.7273583Z "config": { 2023-11-14T04:48:17.7274141Z "mediaType": "application/vnd.docker.container.image.v1+json", 2023-11-14T04:48:17.7274820Z "size": 44107, 2023-11-14T04:48:17.7275495Z "digest": "sha256:2e9cd9f64120afd8fb911e301ea6adc44cad71e8bfde2785095ab22e8b7f1f32" 2023-11-14T04:48:17.7276293Z }, 2023-11-14T04:48:17.7279164Z "layers": [ 2023-11-14T04:48:17.7279621Z { 2023-11-14T04:48:17.7280617Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7281871Z "size": 28580519, 2023-11-14T04:48:17.7283178Z "digest": "sha256:f0412dfb1aaea4892c13dab6f771bcd79641a4bdcb521ce881f5dcfc0df9a7a5" 2023-11-14T04:48:17.7284604Z }, 2023-11-14T04:48:17.7285386Z { 2023-11-14T04:48:17.7286545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7287318Z "size": 7943208, 2023-11-14T04:48:17.7288001Z "digest": "sha256:20d547ab5eb53128ec992abaae1d1f5a15bfe39d7a18ac0e5e265b383ebf71bc" 2023-11-14T04:48:17.7288785Z }, 2023-11-14T04:48:17.7289082Z { 2023-11-14T04:48:17.7289597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7290290Z "size": 56226342, 2023-11-14T04:48:17.7291594Z "digest": "sha256:ece84004a3cd8e7689a2ff0b1a1856a941e6cc31f7ec1eedc39073035a7adddd" 2023-11-14T04:48:17.7292374Z }, 2023-11-14T04:48:17.7292684Z { 2023-11-14T04:48:17.7293219Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7293894Z "size": 186, 2023-11-14T04:48:17.7294586Z "digest": "sha256:b08eef4b90c81de27f73f4aa3522ac14ee1e1146db910e97e17d9d25a969a292" 2023-11-14T04:48:17.7295367Z }, 2023-11-14T04:48:17.7295647Z { 2023-11-14T04:48:17.7296181Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7296870Z "size": 6879, 2023-11-14T04:48:17.7297535Z "digest": "sha256:aa315b7808f0b5e5badf02398e1f7ce1780952eda559defa9fb204e4f19c54c4" 2023-11-14T04:48:17.7298320Z }, 2023-11-14T04:48:17.7298615Z { 2023-11-14T04:48:17.7299129Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7299824Z "size": 1383661429, 2023-11-14T04:48:17.7300512Z "digest": "sha256:3288913d5f128020c71260b4e042a0fdda1ccf5966b3d1c9ef205a5099796a74" 2023-11-14T04:48:17.7301249Z }, 2023-11-14T04:48:17.7301543Z { 2023-11-14T04:48:17.7302315Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7302987Z "size": 62266, 2023-11-14T04:48:17.7303676Z "digest": "sha256:afae2e58c21592a24791ecc7c032f4eb2a1f279ebc922dd3f5d061f62ccb27ce" 2023-11-14T04:48:17.7304461Z }, 2023-11-14T04:48:17.7304739Z { 2023-11-14T04:48:17.7305272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7305961Z "size": 1679, 2023-11-14T04:48:17.7306604Z "digest": "sha256:52a5515aef88a30087855f1aea16480c9b3f88f5b596e2575d95f805fa827a66" 2023-11-14T04:48:17.7307361Z }, 2023-11-14T04:48:17.7307659Z { 2023-11-14T04:48:17.7308173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7308861Z "size": 1522, 2023-11-14T04:48:17.7309541Z "digest": "sha256:fc5398be2cb5046540483506f64caad54db6a0c992ddd1ac93b179c04d2566e5" 2023-11-14T04:48:17.7310294Z }, 2023-11-14T04:48:17.7310592Z { 2023-11-14T04:48:17.7311122Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7311807Z "size": 2445828736, 2023-11-14T04:48:17.7312500Z "digest": "sha256:c868322d41012c4c8c7a47807ec9da5a73e6fba9d43508f958818fedecfdd7d0" 2023-11-14T04:48:17.7313269Z }, 2023-11-14T04:48:17.7313687Z { 2023-11-14T04:48:17.7314244Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7314937Z "size": 85863, 2023-11-14T04:48:17.7315599Z "digest": "sha256:77c5d5a121d582b9bed76ba030514196fc1f7fa0a97d644dd9f48be27fc44030" 2023-11-14T04:48:17.7316369Z }, 2023-11-14T04:48:17.7316666Z { 2023-11-14T04:48:17.7317176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7317872Z "size": 1802, 2023-11-14T04:48:17.7318543Z "digest": "sha256:742ea107d5593f5832e73f71fcd75de0d497ebd6aa3b52731ae9ff645f7c8dfa" 2023-11-14T04:48:17.7319296Z }, 2023-11-14T04:48:17.7319594Z { 2023-11-14T04:48:17.7320126Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7320810Z "size": 703101545, 2023-11-14T04:48:17.7321508Z "digest": "sha256:ea54d9137d4eaa49d45e879985d46d37927eb6888ed76ac84c44bbea18aa5214" 2023-11-14T04:48:17.7322276Z }, 2023-11-14T04:48:17.7322554Z { 2023-11-14T04:48:17.7323088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7323778Z "size": 544, 2023-11-14T04:48:17.7324428Z "digest": "sha256:34d6327ee799ef8cfce6ce61681fdc8857945693f582a0f802ef484078d1d4fe" 2023-11-14T04:48:17.7325193Z }, 2023-11-14T04:48:17.7325490Z { 2023-11-14T04:48:17.7326001Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7326689Z "size": 1275, 2023-11-14T04:48:17.7327354Z "digest": "sha256:2f5ab0288987c559a6be7efb54eb348273e5fd656728862f147635163bd90195" 2023-11-14T04:48:17.7328099Z }, 2023-11-14T04:48:17.7328394Z { 2023-11-14T04:48:17.7328922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7329601Z "size": 483, 2023-11-14T04:48:17.7330268Z "digest": "sha256:5e730d5fd91ad45c18d388d8678881deb27a599a37d8f5bafe8f8e9260c52074" 2023-11-14T04:48:17.7331538Z }, 2023-11-14T04:48:17.7331824Z { 2023-11-14T04:48:17.7332371Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7333067Z "size": 91702437, 2023-11-14T04:48:17.7333726Z "digest": "sha256:2020052b174f67659c6ad7b917a5c66ac34e08371e50f056bf55efb5d7070b49" 2023-11-14T04:48:17.7334480Z }, 2023-11-14T04:48:17.7334776Z { 2023-11-14T04:48:17.7335287Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7335978Z "size": 2999, 2023-11-14T04:48:17.7336641Z "digest": "sha256:80f0c3756826aaf127c22986788cf834fa34bde763927f4fffae89e27d26d918" 2023-11-14T04:48:17.7337377Z }, 2023-11-14T04:48:17.7337672Z { 2023-11-14T04:48:17.7338201Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7338872Z "size": 1812, 2023-11-14T04:48:17.7339700Z "digest": "sha256:5cb4c5d69233695279a8806cf1493a8cd477a9dc5e5990a1ecba05e1eb72c4f1" 2023-11-14T04:48:17.7340466Z }, 2023-11-14T04:48:17.7340740Z { 2023-11-14T04:48:17.7341279Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7341968Z "size": 701, 2023-11-14T04:48:17.7342608Z "digest": "sha256:192c8af52128c5185bdfc54e509898f55fcdd6b549c0776386a4339c89c8bdd2" 2023-11-14T04:48:17.7343371Z }, 2023-11-14T04:48:17.7343666Z { 2023-11-14T04:48:17.7344178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7344869Z "size": 2447950741, 2023-11-14T04:48:17.7345578Z "digest": "sha256:fac5f79d8f40abb579dd6eb7b36f53158f64357b1d6c62014051d84db330407f" 2023-11-14T04:48:17.7346353Z }, 2023-11-14T04:48:17.7346632Z { 2023-11-14T04:48:17.7347163Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7347853Z "size": 380, 2023-11-14T04:48:17.7348490Z "digest": "sha256:ac2c08198bc746ba4dd1830557c7357375c3243e14783b9477f80d10debdbe68" 2023-11-14T04:48:17.7349258Z }, 2023-11-14T04:48:17.7349555Z { 2023-11-14T04:48:17.7350068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7350872Z "size": 6320, 2023-11-14T04:48:17.7351575Z "digest": "sha256:54c34a5c940ffcc48f1786028cd3b73e2b5861a3ef9eef9dc8fd7a6f2949e2b5" 2023-11-14T04:48:17.7352330Z }, 2023-11-14T04:48:17.7352623Z { 2023-11-14T04:48:17.7353153Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7353826Z "size": 802, 2023-11-14T04:48:17.7354493Z "digest": "sha256:c597c95fbef4a958bd6184477f9fb135ea98103919dbe55a4e6ab6b406fd3fbf" 2023-11-14T04:48:17.7355262Z }, 2023-11-14T04:48:17.7355539Z { 2023-11-14T04:48:17.7356065Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7356755Z "size": 106, 2023-11-14T04:48:17.7357401Z "digest": "sha256:3808c65d8c0f541cbeb2bb89123651d4fea349565b734464c125ef27df379d84" 2023-11-14T04:48:17.7358171Z }, 2023-11-14T04:48:17.7358464Z { 2023-11-14T04:48:17.7358977Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7359664Z "size": 887, 2023-11-14T04:48:17.7360354Z "digest": "sha256:c715fe586a58db9e4bbf1fd8c7d3a8e7618a20681fb03a15391f8e6e9d0874d1" 2023-11-14T04:48:17.7361116Z }, 2023-11-14T04:48:17.7361412Z { 2023-11-14T04:48:17.7361939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7362618Z "size": 121477383, 2023-11-14T04:48:17.7363326Z "digest": "sha256:1f385a3986c3b6edfc7c0a892c30cfda8ffd49d10bdae2d86142103c78e346e5" 2023-11-14T04:48:17.7364102Z }, 2023-11-14T04:48:17.7364380Z { 2023-11-14T04:48:17.7364904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7365590Z "size": 109, 2023-11-14T04:48:17.7366231Z "digest": "sha256:d8ad661dea2125376bf054839466f30a001f2e348e54fe2951b0bcf5875a5582" 2023-11-14T04:48:17.7367004Z }, 2023-11-14T04:48:17.7367302Z { 2023-11-14T04:48:17.7367814Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7368506Z "size": 559, 2023-11-14T04:48:17.7369156Z "digest": "sha256:7195438c44c18d6054779424c46121854fdf1ca160e56cf544f828e4a06b1daf" 2023-11-14T04:48:17.7369889Z }, 2023-11-14T04:48:17.7370190Z { 2023-11-14T04:48:17.7371117Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7371886Z "size": 1176144, 2023-11-14T04:48:17.7372591Z "digest": "sha256:3c5a16afcc5fb51b7d68a3adff72d95957eebfe9a7d1250f676b833192fae795" 2023-11-14T04:48:17.7373378Z }, 2023-11-14T04:48:17.7373660Z { 2023-11-14T04:48:17.7374190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7374887Z "size": 103, 2023-11-14T04:48:17.7375534Z "digest": "sha256:23ecc86c3805726132ad4de0755e2e1b4b67b288a4c275ee768335e02532353b" 2023-11-14T04:48:17.7376299Z }, 2023-11-14T04:48:17.7376595Z { 2023-11-14T04:48:17.7377323Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7378019Z "size": 1489, 2023-11-14T04:48:17.7378703Z "digest": "sha256:54f5b07b5ce6d502f30fb982b02be02b8a98e060cd57201966010e0e6b5045a7" 2023-11-14T04:48:17.7379457Z }, 2023-11-14T04:48:17.7379754Z { 2023-11-14T04:48:17.7380287Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7380962Z "size": 437057585, 2023-11-14T04:48:17.7381663Z "digest": "sha256:e8400b0ad3ec1329a7bd5a0e8f517c90acc59ea28ed92e1935a5065dd4b35df4" 2023-11-14T04:48:17.7382439Z }, 2023-11-14T04:48:17.7382717Z { 2023-11-14T04:48:17.7383246Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7383931Z "size": 160, 2023-11-14T04:48:17.7384578Z "digest": "sha256:7ef1ab3712ec09d84d8276812432df58573eb5a4f5091c247e9b4e25b6af87c6" 2023-11-14T04:48:17.7385339Z }, 2023-11-14T04:48:17.7385636Z { 2023-11-14T04:48:17.7386150Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7386849Z "size": 452, 2023-11-14T04:48:17.7387509Z "digest": "sha256:51783a9a2a4aa2c18cbe4d45228485f49089300d9d197432d086ecaee74af1ad" 2023-11-14T04:48:17.7388248Z }, 2023-11-14T04:48:17.7388660Z { 2023-11-14T04:48:17.7389210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7389887Z "size": 56590734, 2023-11-14T04:48:17.7390570Z "digest": "sha256:0778288efbc7d96e3abf251814c144df31f43041202d85b61f621d86575bbd6b" 2023-11-14T04:48:17.7391332Z }, 2023-11-14T04:48:17.7391613Z { 2023-11-14T04:48:17.7392142Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7392836Z "size": 104, 2023-11-14T04:48:17.7393491Z "digest": "sha256:8b7bba61649c14d9c31c57ee9f8fa23bf80a3d67f18e41227207fa66991b6ac8" 2023-11-14T04:48:17.7394261Z }, 2023-11-14T04:48:17.7394557Z { 2023-11-14T04:48:17.7395068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7395767Z "size": 402, 2023-11-14T04:48:17.7396447Z "digest": "sha256:d913ee57fbe5fa4af8feec010598662835bc5df4d9e5af62be2433f2d9ad6aed" 2023-11-14T04:48:17.7397212Z }, 2023-11-14T04:48:17.7397510Z { 2023-11-14T04:48:17.7398044Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7398719Z "size": 20262063, 2023-11-14T04:48:17.7399421Z "digest": "sha256:ad4cadf638a3dafe145549809d6ff107dd1e75fc67391c130eb3a2d3fecd0505" 2023-11-14T04:48:17.7400204Z }, 2023-11-14T04:48:17.7400483Z { 2023-11-14T04:48:17.7401009Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7401694Z "size": 440, 2023-11-14T04:48:17.7402354Z "digest": "sha256:88ffa4df8050fbf76da69f38a234218b38e293d0e0fc07c4c7c3cadbb5181c71" 2023-11-14T04:48:17.7403128Z }, 2023-11-14T04:48:17.7403424Z { 2023-11-14T04:48:17.7403938Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7404640Z "size": 701, 2023-11-14T04:48:17.7405299Z "digest": "sha256:192c8af52128c5185bdfc54e509898f55fcdd6b549c0776386a4339c89c8bdd2" 2023-11-14T04:48:17.7406039Z }, 2023-11-14T04:48:17.7406335Z { 2023-11-14T04:48:17.7406868Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7407541Z "size": 140, 2023-11-14T04:48:17.7408209Z "digest": "sha256:7b66c65e46fc5f6f1b5820e15fbe4297e3c649284760f9d27ee4a43cf7927757" 2023-11-14T04:48:17.7408973Z }, 2023-11-14T04:48:17.7409250Z { 2023-11-14T04:48:17.7409780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7410469Z "size": 137, 2023-11-14T04:48:17.7411595Z "digest": "sha256:ca20e2be29876a691955ecb5b6730649f660a1f08ac903aebd73332a8442b002" 2023-11-14T04:48:17.7412368Z }, 2023-11-14T04:48:17.7412665Z { 2023-11-14T04:48:17.7413176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7413863Z "size": 189, 2023-11-14T04:48:17.7414699Z "digest": "sha256:cb0f483351b235bd9f8e8148798ae79f5e9c3a9875fcee4c94836530dea0db07" 2023-11-14T04:48:17.7415446Z }, 2023-11-14T04:48:17.7415742Z { 2023-11-14T04:48:17.7416272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7416954Z "size": 563, 2023-11-14T04:48:17.7417638Z "digest": "sha256:034bfa5626bffed7d19af0b036fe4a772abb0c2fb625f9a2b7208f90083da374" 2023-11-14T04:48:17.7418421Z }, 2023-11-14T04:48:17.7418698Z { 2023-11-14T04:48:17.7419228Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7419922Z "size": 43166336, 2023-11-14T04:48:17.7420592Z "digest": "sha256:c1310534828e57dc483e3b3dbf61df39e6bf2706423a53f15f6d165bd6da4ba8" 2023-11-14T04:48:17.7421393Z }, 2023-11-14T04:48:17.7421693Z { 2023-11-14T04:48:17.7422202Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7422892Z "size": 106, 2023-11-14T04:48:17.7423551Z "digest": "sha256:2aa9bf51637bd210c796735975c6f8b5a890a3d19f0595a9b082d11f34e7f713" 2023-11-14T04:48:17.7424300Z }, 2023-11-14T04:48:17.7424597Z { 2023-11-14T04:48:17.7425129Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7425918Z "size": 1154, 2023-11-14T04:48:17.7426621Z "digest": "sha256:f9bcb4185c0a07a32c8ac0f2c81c3998c78fc6e80d28eb4f17d1624dde28f8fe" 2023-11-14T04:48:17.7427405Z }, 2023-11-14T04:48:17.7427682Z { 2023-11-14T04:48:17.7428211Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7428902Z "size": 701, 2023-11-14T04:48:17.7429546Z "digest": "sha256:192c8af52128c5185bdfc54e509898f55fcdd6b549c0776386a4339c89c8bdd2" 2023-11-14T04:48:17.7430304Z }, 2023-11-14T04:48:17.7430601Z { 2023-11-14T04:48:17.7431115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7431800Z "size": 138, 2023-11-14T04:48:17.7432470Z "digest": "sha256:c70de30968b07031424eeb3b61dd9d598a58272e2831f2230ec1bcb25ab20f6e" 2023-11-14T04:48:17.7433231Z }, 2023-11-14T04:48:17.7433527Z { 2023-11-14T04:48:17.7434059Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7434732Z "size": 118, 2023-11-14T04:48:17.7435408Z "digest": "sha256:1ad45855bb002887803eb27e45e0e99c3ce1ddb2d99dcbf2c8a9476c4482b406" 2023-11-14T04:48:17.7436180Z }, 2023-11-14T04:48:17.7436455Z { 2023-11-14T04:48:17.7436983Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7437678Z "size": 1046484338, 2023-11-14T04:48:17.7438372Z "digest": "sha256:9882884fae03caa7dcebd2e2f82e96a4f7fcf0091edfe59c469e73b7e191ece1" 2023-11-14T04:48:17.7439153Z }, 2023-11-14T04:48:17.7439449Z { 2023-11-14T04:48:17.7439961Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7440650Z "size": 172, 2023-11-14T04:48:17.7441320Z "digest": "sha256:ef5e32cf3412ce34dfa983a36d1dc538b162222f661f9d304117a3e93065d9a9" 2023-11-14T04:48:17.7442066Z }, 2023-11-14T04:48:17.7442368Z { 2023-11-14T04:48:17.7442894Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7443566Z "size": 1840, 2023-11-14T04:48:17.7444268Z "digest": "sha256:cdf3b334dcb11e7aab89d40de4889eb4e7ea9998cedd6ef890ca38a48c45b8d1" 2023-11-14T04:48:17.7445064Z }, 2023-11-14T04:48:17.7445340Z { 2023-11-14T04:48:17.7445870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7446562Z "size": 7529791, 2023-11-14T04:48:17.7447246Z "digest": "sha256:ba019cce1eadffd12abe55f725b19e0c87b222ce899d0b66d30531601b152a98" 2023-11-14T04:48:17.7448033Z }, 2023-11-14T04:48:17.7448328Z { 2023-11-14T04:48:17.7448839Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7449532Z "size": 163, 2023-11-14T04:48:17.7450192Z "digest": "sha256:b59f1e2807d4563379c56ed6f4f0354b92b58465efcaa63deb40ee9737ef7e49" 2023-11-14T04:48:17.7451401Z }, 2023-11-14T04:48:17.7451703Z { 2023-11-14T04:48:17.7452391Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7453065Z "size": 7940, 2023-11-14T04:48:17.7453759Z "digest": "sha256:4dce0aae7c66ff5572e6f7ac9b1c66af607aade58e0e3ff0215edb344f52389f" 2023-11-14T04:48:17.7454558Z }, 2023-11-14T04:48:17.7454835Z { 2023-11-14T04:48:17.7455366Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7456065Z "size": 8071, 2023-11-14T04:48:17.7456729Z "digest": "sha256:93c6911b02879dc3f1578acef87dc6faea15c8d288cdb6ad21e5789a8f36af0c" 2023-11-14T04:48:17.7457502Z }, 2023-11-14T04:48:17.7457796Z { 2023-11-14T04:48:17.7458312Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7459002Z "size": 304, 2023-11-14T04:48:17.7459684Z "digest": "sha256:fcdaac0fbb69c12f1e7245a397b7ec568e12226a94726853e85fbd19d3d78fe7" 2023-11-14T04:48:17.7460443Z }, 2023-11-14T04:48:17.7460738Z { 2023-11-14T04:48:17.7461268Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7461949Z "size": 7614964, 2023-11-14T04:48:17.7462636Z "digest": "sha256:d80b8d0f01dcea7652b8d14f9705e86811d00a8fe09dac606c4b3e815a865fcb" 2023-11-14T04:48:17.7463405Z }, 2023-11-14T04:48:17.7463801Z { 2023-11-14T04:48:17.7464357Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7465049Z "size": 108, 2023-11-14T04:48:17.7465683Z "digest": "sha256:762352f3c28b2180f18b6245538bc16e3e52af45829072df88c6e4ab202eeddc" 2023-11-14T04:48:17.7466435Z }, 2023-11-14T04:48:17.7466765Z + exit 0 2023-11-14T04:48:17.7467082Z { 2023-11-14T04:48:17.7467595Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7468292Z "size": 54145478, 2023-11-14T04:48:17.7468992Z "digest": "sha256:f39a9cff48d63e07c1065b34be0d3815db7edd7292a1ea820a3a48ad62781f26" 2023-11-14T04:48:17.7469749Z }, 2023-11-14T04:48:17.7470044Z { 2023-11-14T04:48:17.7470574Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7471252Z "size": 513, 2023-11-14T04:48:17.7471923Z "digest": "sha256:0c4fbc80353d21fa656812cf7eb51b983b9f8513d020d52c8da372a08897ee39" 2023-11-14T04:48:17.7472687Z }, 2023-11-14T04:48:17.7472969Z { 2023-11-14T04:48:17.7473501Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7474193Z "size": 2997737455, 2023-11-14T04:48:17.7474892Z "digest": "sha256:bb1924fd13d8efdb4bc768ada19376b6b89d094b36dd6e624d81e54a8ed37c55" 2023-11-14T04:48:17.7475676Z }, 2023-11-14T04:48:17.7475969Z { 2023-11-14T04:48:17.7476480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:17.7477221Z "size": 106, 2023-11-14T04:48:17.7477891Z "digest": "sha256:ea619973e9afb03f2d49a961986c66cf734bd9243e567cd5acd9818fd418ef49" 2023-11-14T04:48:17.7478642Z } 2023-11-14T04:48:17.7478938Z ] 2023-11-14T04:48:17.7479229Z } 2023-11-14T04:48:17.7670973Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2023-11-14T04:48:17.7671736Z with: 2023-11-14T04:48:17.7672899Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.7674186Z env: 2023-11-14T04:48:17.7674480Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:17.7674868Z ##[endgroup] 2023-11-14T04:48:17.7699129Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2023-11-14T04:48:17.7699982Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2023-11-14T04:48:17.7700840Z # ignore output since only exit code is used for conditional 2023-11-14T04:48:17.7701718Z # only pull docker image if it's not available locally 2023-11-14T04:48:17.7702659Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2023-11-14T04:48:17.7703592Z  retry docker pull "${DOCKER_IMAGE}" 2023-11-14T04:48:17.7704123Z fi 2023-11-14T04:48:17.7712446Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:17.7713064Z env: 2023-11-14T04:48:17.7713397Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:17.7714663Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:17.7715987Z ##[endgroup] 2023-11-14T04:48:18.0183718Z 3cbebab0ba3af5b92c112f499347ddfdd0f631c8: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9 2023-11-14T04:48:18.0184865Z f0412dfb1aae: Pulling fs layer 2023-11-14T04:48:18.0185448Z 20d547ab5eb5: Pulling fs layer 2023-11-14T04:48:18.0185976Z ece84004a3cd: Pulling fs layer 2023-11-14T04:48:18.0186522Z b08eef4b90c8: Pulling fs layer 2023-11-14T04:48:18.0187060Z aa315b7808f0: Pulling fs layer 2023-11-14T04:48:18.0187593Z 3288913d5f12: Pulling fs layer 2023-11-14T04:48:18.0188155Z afae2e58c215: Pulling fs layer 2023-11-14T04:48:18.0188701Z 52a5515aef88: Pulling fs layer 2023-11-14T04:48:18.0189233Z fc5398be2cb5: Pulling fs layer 2023-11-14T04:48:18.0189753Z b08eef4b90c8: Waiting 2023-11-14T04:48:18.0190234Z c868322d4101: Pulling fs layer 2023-11-14T04:48:18.0190749Z 77c5d5a121d5: Pulling fs layer 2023-11-14T04:48:18.0191276Z 742ea107d559: Pulling fs layer 2023-11-14T04:48:18.0191790Z aa315b7808f0: Waiting 2023-11-14T04:48:18.0192233Z 3288913d5f12: Waiting 2023-11-14T04:48:18.0192690Z afae2e58c215: Waiting 2023-11-14T04:48:18.0193172Z ea54d9137d4e: Pulling fs layer 2023-11-14T04:48:18.0193688Z 34d6327ee799: Pulling fs layer 2023-11-14T04:48:18.0194223Z 2f5ab0288987: Pulling fs layer 2023-11-14T04:48:18.0194753Z 5e730d5fd91a: Pulling fs layer 2023-11-14T04:48:18.0195266Z 2020052b174f: Pulling fs layer 2023-11-14T04:48:18.0206477Z 80f0c3756826: Pulling fs layer 2023-11-14T04:48:18.0207107Z 77c5d5a121d5: Waiting 2023-11-14T04:48:18.0208118Z c868322d4101: Waiting 2023-11-14T04:48:18.0208847Z 5cb4c5d69233: Pulling fs layer 2023-11-14T04:48:18.0209394Z 192c8af52128: Pulling fs layer 2023-11-14T04:48:18.0209903Z 52a5515aef88: Waiting 2023-11-14T04:48:18.0210385Z fac5f79d8f40: Pulling fs layer 2023-11-14T04:48:18.0211353Z ea54d9137d4e: Waiting 2023-11-14T04:48:18.0211806Z 2f5ab0288987: Waiting 2023-11-14T04:48:18.0212264Z 742ea107d559: Waiting 2023-11-14T04:48:18.0212717Z 34d6327ee799: Waiting 2023-11-14T04:48:18.0213159Z fc5398be2cb5: Waiting 2023-11-14T04:48:18.0213642Z ac2c08198bc7: Pulling fs layer 2023-11-14T04:48:18.0214317Z 80f0c3756826: Waiting 2023-11-14T04:48:18.0215330Z 54c34a5c940f: Pulling fs layer 2023-11-14T04:48:18.0216201Z c597c95fbef4: Pulling fs layer 2023-11-14T04:48:18.0216732Z ac2c08198bc7: Waiting 2023-11-14T04:48:18.0217192Z 3808c65d8c0f: Pulling fs layer 2023-11-14T04:48:18.0217708Z 54c34a5c940f: Waiting 2023-11-14T04:48:18.0218182Z c715fe586a58: Pulling fs layer 2023-11-14T04:48:18.0218700Z 1f385a3986c3: Pulling fs layer 2023-11-14T04:48:18.0219523Z 3808c65d8c0f: Waiting 2023-11-14T04:48:18.0220444Z c715fe586a58: Waiting 2023-11-14T04:48:18.0221330Z c597c95fbef4: Waiting 2023-11-14T04:48:18.0221845Z d8ad661dea21: Pulling fs layer 2023-11-14T04:48:18.0222378Z 7195438c44c1: Pulling fs layer 2023-11-14T04:48:18.0222870Z 192c8af52128: Waiting 2023-11-14T04:48:18.0223349Z 3c5a16afcc5f: Pulling fs layer 2023-11-14T04:48:18.0223884Z 23ecc86c3805: Pulling fs layer 2023-11-14T04:48:18.0224375Z 7195438c44c1: Waiting 2023-11-14T04:48:18.0224828Z 1f385a3986c3: Waiting 2023-11-14T04:48:18.0225284Z d8ad661dea21: Waiting 2023-11-14T04:48:18.0225728Z fac5f79d8f40: Waiting 2023-11-14T04:48:18.0226229Z 3c5a16afcc5f: Waiting 2023-11-14T04:48:18.0226716Z 54f5b07b5ce6: Pulling fs layer 2023-11-14T04:48:18.0227235Z e8400b0ad3ec: Pulling fs layer 2023-11-14T04:48:18.0227770Z 7ef1ab3712ec: Pulling fs layer 2023-11-14T04:48:18.0228305Z 51783a9a2a4a: Pulling fs layer 2023-11-14T04:48:18.0228796Z 23ecc86c3805: Waiting 2023-11-14T04:48:18.0229268Z 0778288efbc7: Pulling fs layer 2023-11-14T04:48:18.0229779Z 54f5b07b5ce6: Waiting 2023-11-14T04:48:18.0230247Z 8b7bba61649c: Pulling fs layer 2023-11-14T04:48:18.0230944Z e8400b0ad3ec: Waiting 2023-11-14T04:48:18.0231422Z d913ee57fbe5: Pulling fs layer 2023-11-14T04:48:18.0231924Z 51783a9a2a4a: Waiting 2023-11-14T04:48:18.0232402Z ad4cadf638a3: Pulling fs layer 2023-11-14T04:48:18.0232943Z 88ffa4df8050: Pulling fs layer 2023-11-14T04:48:18.0233447Z 7ef1ab3712ec: Waiting 2023-11-14T04:48:18.0233926Z 7b66c65e46fc: Pulling fs layer 2023-11-14T04:48:18.0234442Z 0778288efbc7: Waiting 2023-11-14T04:48:18.0234899Z ca20e2be2987: Pulling fs layer 2023-11-14T04:48:18.0235433Z cb0f483351b2: Pulling fs layer 2023-11-14T04:48:18.0235967Z 034bfa5626bf: Pulling fs layer 2023-11-14T04:48:18.0236464Z 88ffa4df8050: Waiting 2023-11-14T04:48:18.0236922Z 7b66c65e46fc: Waiting 2023-11-14T04:48:18.0237394Z c1310534828e: Pulling fs layer 2023-11-14T04:48:18.0237886Z cb0f483351b2: Waiting 2023-11-14T04:48:18.0238342Z 8b7bba61649c: Waiting 2023-11-14T04:48:18.0238818Z 2aa9bf51637b: Pulling fs layer 2023-11-14T04:48:18.0239321Z c1310534828e: Waiting 2023-11-14T04:48:18.0239782Z d913ee57fbe5: Waiting 2023-11-14T04:48:18.0240261Z f9bcb4185c0a: Pulling fs layer 2023-11-14T04:48:18.0240764Z 034bfa5626bf: Waiting 2023-11-14T04:48:18.0241239Z c70de30968b0: Pulling fs layer 2023-11-14T04:48:18.0241771Z 1ad45855bb00: Pulling fs layer 2023-11-14T04:48:18.0242284Z 9882884fae03: Pulling fs layer 2023-11-14T04:48:18.0242798Z ad4cadf638a3: Waiting 2023-11-14T04:48:18.0243282Z ef5e32cf3412: Pulling fs layer 2023-11-14T04:48:18.0243783Z 2aa9bf51637b: Waiting 2023-11-14T04:48:18.0244240Z c70de30968b0: Waiting 2023-11-14T04:48:18.0244699Z cdf3b334dcb1: Pulling fs layer 2023-11-14T04:48:18.0245214Z 9882884fae03: Waiting 2023-11-14T04:48:18.0245697Z ba019cce1ead: Pulling fs layer 2023-11-14T04:48:18.0246200Z ef5e32cf3412: Waiting 2023-11-14T04:48:18.0246674Z b59f1e2807d4: Pulling fs layer 2023-11-14T04:48:18.0247213Z 4dce0aae7c66: Pulling fs layer 2023-11-14T04:48:18.0247729Z 93c6911b0287: Pulling fs layer 2023-11-14T04:48:18.0248270Z fcdaac0fbb69: Pulling fs layer 2023-11-14T04:48:18.0248796Z 4dce0aae7c66: Waiting 2023-11-14T04:48:18.0249244Z 93c6911b0287: Waiting 2023-11-14T04:48:18.0249723Z d80b8d0f01dc: Pulling fs layer 2023-11-14T04:48:18.0250256Z 762352f3c28b: Pulling fs layer 2023-11-14T04:48:18.0250979Z fcdaac0fbb69: Waiting 2023-11-14T04:48:18.0251484Z f39a9cff48d6: Pulling fs layer 2023-11-14T04:48:18.0252021Z 0c4fbc80353d: Pulling fs layer 2023-11-14T04:48:18.0252538Z bb1924fd13d8: Pulling fs layer 2023-11-14T04:48:18.0253072Z ea619973e9af: Pulling fs layer 2023-11-14T04:48:18.0253585Z 0c4fbc80353d: Waiting 2023-11-14T04:48:18.0254025Z bb1924fd13d8: Waiting 2023-11-14T04:48:18.2789178Z 20d547ab5eb5: Verifying Checksum 2023-11-14T04:48:18.2789700Z 20d547ab5eb5: Download complete 2023-11-14T04:48:18.3667517Z b08eef4b90c8: Verifying Checksum 2023-11-14T04:48:18.3668053Z b08eef4b90c8: Download complete 2023-11-14T04:48:18.4636103Z aa315b7808f0: Download complete 2023-11-14T04:48:19.1713509Z f0412dfb1aae: Verifying Checksum 2023-11-14T04:48:19.1714358Z f0412dfb1aae: Download complete 2023-11-14T04:48:19.2475407Z afae2e58c215: Download complete 2023-11-14T04:48:19.3262981Z 52a5515aef88: Verifying Checksum 2023-11-14T04:48:19.3263538Z 52a5515aef88: Download complete 2023-11-14T04:48:19.4378953Z fc5398be2cb5: Verifying Checksum 2023-11-14T04:48:19.4379498Z fc5398be2cb5: Download complete 2023-11-14T04:48:19.9717366Z f0412dfb1aae: Pull complete 2023-11-14T04:48:20.2484313Z 20d547ab5eb5: Pull complete 2023-11-14T04:48:20.9002532Z ece84004a3cd: Verifying Checksum 2023-11-14T04:48:20.9003200Z ece84004a3cd: Download complete 2023-11-14T04:48:21.0036227Z 77c5d5a121d5: Download complete 2023-11-14T04:48:21.1061930Z 742ea107d559: Verifying Checksum 2023-11-14T04:48:21.1062566Z 742ea107d559: Download complete 2023-11-14T04:48:21.9481338Z ece84004a3cd: Pull complete 2023-11-14T04:48:22.0542812Z b08eef4b90c8: Pull complete 2023-11-14T04:48:22.1625500Z aa315b7808f0: Pull complete 2023-11-14T04:48:34.5850534Z ea54d9137d4e: Verifying Checksum 2023-11-14T04:48:34.5851544Z ea54d9137d4e: Download complete 2023-11-14T04:48:34.6737513Z 34d6327ee799: Verifying Checksum 2023-11-14T04:48:34.6738140Z 34d6327ee799: Download complete 2023-11-14T04:48:34.7482563Z 2f5ab0288987: Download complete 2023-11-14T04:48:34.8292830Z 5e730d5fd91a: Verifying Checksum 2023-11-14T04:48:34.8293428Z 5e730d5fd91a: Download complete 2023-11-14T04:48:37.0403201Z 2020052b174f: Verifying Checksum 2023-11-14T04:48:37.0403870Z 2020052b174f: Download complete 2023-11-14T04:48:37.1316031Z 80f0c3756826: Verifying Checksum 2023-11-14T04:48:37.1316677Z 80f0c3756826: Download complete 2023-11-14T04:48:37.2146995Z 5cb4c5d69233: Download complete 2023-11-14T04:48:37.3232706Z 192c8af52128: Download complete 2023-11-14T04:49:10.4774909Z 3288913d5f12: Verifying Checksum 2023-11-14T04:49:10.4775569Z 3288913d5f12: Download complete 2023-11-14T04:49:10.6081024Z ac2c08198bc7: Verifying Checksum 2023-11-14T04:49:10.6082017Z ac2c08198bc7: Download complete 2023-11-14T04:49:10.6873147Z 54c34a5c940f: Verifying Checksum 2023-11-14T04:49:10.6873806Z 54c34a5c940f: Download complete 2023-11-14T04:49:10.7731741Z c597c95fbef4: Verifying Checksum 2023-11-14T04:49:10.7732426Z c597c95fbef4: Download complete 2023-11-14T04:49:10.8637912Z 3808c65d8c0f: Verifying Checksum 2023-11-14T04:49:10.8638579Z 3808c65d8c0f: Download complete 2023-11-14T04:49:10.9481175Z c715fe586a58: Verifying Checksum 2023-11-14T04:49:10.9482378Z c715fe586a58: Download complete 2023-11-14T04:49:14.1733040Z 1f385a3986c3: Verifying Checksum 2023-11-14T04:49:14.1733664Z 1f385a3986c3: Download complete 2023-11-14T04:49:14.2620784Z d8ad661dea21: Download complete 2023-11-14T04:49:14.3586987Z 7195438c44c1: Verifying Checksum 2023-11-14T04:49:14.3587598Z 7195438c44c1: Download complete 2023-11-14T04:49:14.5201938Z 3c5a16afcc5f: Verifying Checksum 2023-11-14T04:49:14.5202536Z 3c5a16afcc5f: Download complete 2023-11-14T04:49:14.6092008Z 23ecc86c3805: Verifying Checksum 2023-11-14T04:49:14.6092584Z 23ecc86c3805: Download complete 2023-11-14T04:49:14.6943136Z 54f5b07b5ce6: Verifying Checksum 2023-11-14T04:49:14.6944053Z 54f5b07b5ce6: Download complete 2023-11-14T04:49:23.6529506Z e8400b0ad3ec: Verifying Checksum 2023-11-14T04:49:23.6530587Z e8400b0ad3ec: Download complete 2023-11-14T04:49:23.7540470Z 7ef1ab3712ec: Verifying Checksum 2023-11-14T04:49:23.7541457Z 7ef1ab3712ec: Download complete 2023-11-14T04:49:23.8274021Z 51783a9a2a4a: Download complete 2023-11-14T04:49:25.0586421Z 0778288efbc7: Verifying Checksum 2023-11-14T04:49:25.0587533Z 0778288efbc7: Download complete 2023-11-14T04:49:25.1563415Z 8b7bba61649c: Verifying Checksum 2023-11-14T04:49:25.1564458Z 8b7bba61649c: Download complete 2023-11-14T04:49:25.2624773Z d913ee57fbe5: Verifying Checksum 2023-11-14T04:49:25.2625380Z d913ee57fbe5: Download complete 2023-11-14T04:49:25.5844386Z ad4cadf638a3: Download complete 2023-11-14T04:49:25.6763836Z 88ffa4df8050: Verifying Checksum 2023-11-14T04:49:25.6764426Z 88ffa4df8050: Download complete 2023-11-14T04:49:25.7632688Z 7b66c65e46fc: Verifying Checksum 2023-11-14T04:49:25.7633730Z 7b66c65e46fc: Download complete 2023-11-14T04:49:25.8457880Z ca20e2be2987: Verifying Checksum 2023-11-14T04:49:25.8458483Z ca20e2be2987: Download complete 2023-11-14T04:49:25.9266832Z cb0f483351b2: Verifying Checksum 2023-11-14T04:49:25.9267404Z cb0f483351b2: Download complete 2023-11-14T04:49:26.0062503Z 034bfa5626bf: Verifying Checksum 2023-11-14T04:49:26.0063078Z 034bfa5626bf: Download complete 2023-11-14T04:49:27.2122938Z c1310534828e: Verifying Checksum 2023-11-14T04:49:27.2123552Z c1310534828e: Download complete 2023-11-14T04:49:27.2943750Z 2aa9bf51637b: Verifying Checksum 2023-11-14T04:49:27.2944417Z 2aa9bf51637b: Download complete 2023-11-14T04:49:27.3839686Z f9bcb4185c0a: Verifying Checksum 2023-11-14T04:49:27.3840674Z f9bcb4185c0a: Download complete 2023-11-14T04:49:27.4670084Z c70de30968b0: Verifying Checksum 2023-11-14T04:49:27.4670655Z c70de30968b0: Download complete 2023-11-14T04:49:27.5526906Z 1ad45855bb00: Download complete 2023-11-14T04:49:28.5569795Z fac5f79d8f40: Verifying Checksum 2023-11-14T04:49:28.5570479Z fac5f79d8f40: Download complete 2023-11-14T04:49:28.6447118Z ef5e32cf3412: Verifying Checksum 2023-11-14T04:49:28.6447777Z ef5e32cf3412: Download complete 2023-11-14T04:49:28.7376173Z cdf3b334dcb1: Verifying Checksum 2023-11-14T04:49:28.7376853Z cdf3b334dcb1: Download complete 2023-11-14T04:49:28.9763317Z 3288913d5f12: Pull complete 2023-11-14T04:49:29.0053173Z ba019cce1ead: Verifying Checksum 2023-11-14T04:49:29.0053971Z ba019cce1ead: Download complete 2023-11-14T04:49:29.0934785Z b59f1e2807d4: Verifying Checksum 2023-11-14T04:49:29.0935410Z b59f1e2807d4: Download complete 2023-11-14T04:49:29.1829833Z 4dce0aae7c66: Download complete 2023-11-14T04:49:29.2637159Z 93c6911b0287: Verifying Checksum 2023-11-14T04:49:29.2638212Z 93c6911b0287: Download complete 2023-11-14T04:49:29.3529677Z fcdaac0fbb69: Verifying Checksum 2023-11-14T04:49:29.3530315Z fcdaac0fbb69: Download complete 2023-11-14T04:49:29.6034728Z d80b8d0f01dc: Verifying Checksum 2023-11-14T04:49:29.6035977Z d80b8d0f01dc: Download complete 2023-11-14T04:49:29.6949318Z 762352f3c28b: Verifying Checksum 2023-11-14T04:49:29.6950494Z 762352f3c28b: Download complete 2023-11-14T04:49:30.8559529Z afae2e58c215: Pull complete 2023-11-14T04:49:30.9947160Z f39a9cff48d6: Verifying Checksum 2023-11-14T04:49:30.9947792Z f39a9cff48d6: Download complete 2023-11-14T04:49:31.0776087Z 0c4fbc80353d: Verifying Checksum 2023-11-14T04:49:31.0776669Z 0c4fbc80353d: Download complete 2023-11-14T04:49:32.7043395Z 52a5515aef88: Pull complete 2023-11-14T04:49:34.6958660Z fc5398be2cb5: Pull complete 2023-11-14T04:49:49.8265677Z 9882884fae03: Verifying Checksum 2023-11-14T04:49:49.8266363Z 9882884fae03: Download complete 2023-11-14T04:49:49.9067475Z ea619973e9af: Verifying Checksum 2023-11-14T04:49:49.9068066Z ea619973e9af: Download complete 2023-11-14T04:49:58.4237640Z c868322d4101: Verifying Checksum 2023-11-14T04:49:58.4238221Z c868322d4101: Download complete 2023-11-14T04:50:32.6627832Z bb1924fd13d8: Verifying Checksum 2023-11-14T04:50:32.6628453Z bb1924fd13d8: Download complete 2023-11-14T04:50:32.9362782Z c868322d4101: Pull complete 2023-11-14T04:50:34.6860232Z 77c5d5a121d5: Pull complete 2023-11-14T04:50:36.4770981Z 742ea107d559: Pull complete 2023-11-14T04:50:50.4021877Z ea54d9137d4e: Pull complete 2023-11-14T04:50:52.2816462Z 34d6327ee799: Pull complete 2023-11-14T04:50:54.1604924Z 2f5ab0288987: Pull complete 2023-11-14T04:50:56.0365246Z 5e730d5fd91a: Pull complete 2023-11-14T04:51:00.5239423Z 2020052b174f: Pull complete 2023-11-14T04:51:02.2820370Z 80f0c3756826: Pull complete 2023-11-14T04:51:04.1610743Z 5cb4c5d69233: Pull complete 2023-11-14T04:51:05.8099792Z 192c8af52128: Pull complete 2023-11-14T04:51:46.5454055Z fac5f79d8f40: Pull complete 2023-11-14T04:51:48.4573112Z ac2c08198bc7: Pull complete 2023-11-14T04:51:50.4248006Z 54c34a5c940f: Pull complete 2023-11-14T04:51:52.3037149Z c597c95fbef4: Pull complete 2023-11-14T04:51:54.2079957Z 3808c65d8c0f: Pull complete 2023-11-14T04:51:56.0845785Z c715fe586a58: Pull complete 2023-11-14T04:52:01.9064568Z 1f385a3986c3: Pull complete 2023-11-14T04:52:02.0227393Z d8ad661dea21: Pull complete 2023-11-14T04:52:02.1286321Z 7195438c44c1: Pull complete 2023-11-14T04:52:02.2746696Z 3c5a16afcc5f: Pull complete 2023-11-14T04:52:02.3814786Z 23ecc86c3805: Pull complete 2023-11-14T04:52:02.4854993Z 54f5b07b5ce6: Pull complete 2023-11-14T04:52:12.6490180Z e8400b0ad3ec: Pull complete 2023-11-14T04:52:15.6299805Z 7ef1ab3712ec: Pull complete 2023-11-14T04:52:17.5891815Z 51783a9a2a4a: Pull complete 2023-11-14T04:52:21.3115274Z 0778288efbc7: Pull complete 2023-11-14T04:52:23.6468628Z 8b7bba61649c: Pull complete 2023-11-14T04:52:23.8169588Z d913ee57fbe5: Pull complete 2023-11-14T04:52:24.2293679Z ad4cadf638a3: Pull complete 2023-11-14T04:52:24.3509320Z 88ffa4df8050: Pull complete 2023-11-14T04:52:24.5728574Z 7b66c65e46fc: Pull complete 2023-11-14T04:52:24.6598389Z ca20e2be2987: Pull complete 2023-11-14T04:52:24.7605707Z cb0f483351b2: Pull complete 2023-11-14T04:52:24.8782605Z 034bfa5626bf: Pull complete 2023-11-14T04:52:26.2387566Z c1310534828e: Pull complete 2023-11-14T04:52:26.3342344Z 2aa9bf51637b: Pull complete 2023-11-14T04:52:26.4185590Z f9bcb4185c0a: Pull complete 2023-11-14T04:52:26.6537677Z c70de30968b0: Pull complete 2023-11-14T04:52:26.7684163Z 1ad45855bb00: Pull complete 2023-11-14T04:52:47.5222127Z 9882884fae03: Pull complete 2023-11-14T04:52:47.6415178Z ef5e32cf3412: Pull complete 2023-11-14T04:52:47.7572850Z cdf3b334dcb1: Pull complete 2023-11-14T04:52:48.0361310Z ba019cce1ead: Pull complete 2023-11-14T04:52:48.1571185Z b59f1e2807d4: Pull complete 2023-11-14T04:52:48.2732681Z 4dce0aae7c66: Pull complete 2023-11-14T04:52:48.3931183Z 93c6911b0287: Pull complete 2023-11-14T04:52:48.5038173Z fcdaac0fbb69: Pull complete 2023-11-14T04:52:49.0932101Z d80b8d0f01dc: Pull complete 2023-11-14T04:52:49.1901347Z 762352f3c28b: Pull complete 2023-11-14T04:52:51.0828069Z f39a9cff48d6: Pull complete 2023-11-14T04:52:51.1885591Z 0c4fbc80353d: Pull complete 2023-11-14T04:53:30.2802554Z bb1924fd13d8: Pull complete 2023-11-14T04:53:32.1283502Z ea619973e9af: Pull complete 2023-11-14T04:53:33.4125067Z Digest: sha256:155cc70f81c46492d7369e444f8774cfbc3ff3ec6c7d0217deb03d9c42253389 2023-11-14T04:53:33.9137873Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:53:34.1960062Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:53:34.2175680Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2023-11-14T04:53:34.2176493Z with: 2023-11-14T04:53:34.2176870Z driver-version: 535.54.03 2023-11-14T04:53:34.2177287Z env: 2023-11-14T04:53:34.2177649Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:53:34.2178098Z ##[endgroup] 2023-11-14T04:53:34.3543900Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T04:53:34.3544644Z with: 2023-11-14T04:53:34.3544987Z timeout_minutes: 10 2023-11-14T04:53:34.3545423Z max_attempts: 3 2023-11-14T04:53:34.3586268Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2023-11-14T04:53:34.3627558Z retry_wait_seconds: 10 2023-11-14T04:53:34.3628063Z polling_interval_seconds: 1 2023-11-14T04:53:34.3628555Z warning_on_retry: true 2023-11-14T04:53:34.3629019Z continue_on_error: false 2023-11-14T04:53:34.3629461Z env: 2023-11-14T04:53:34.3629796Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:53:34.3630264Z DRIVER_VERSION: 535.54.03 2023-11-14T04:53:34.3630721Z ##[endgroup] 2023-11-14T04:53:34.4431751Z == Installing nvidia driver NVIDIA-Linux-x86_64-535.54.03.run == 2023-11-14T04:53:34.4433820Z + pre_install_nvidia_driver_amzn2 2023-11-14T04:53:34.4434504Z + sudo yum remove -y nvidia-driver-latest-dkms 2023-11-14T04:53:34.9790552Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-11-14T04:53:35.0384426Z No Match for argument: nvidia-driver-latest-dkms 2023-11-14T04:53:35.0772117Z No Packages marked for removal 2023-11-14T04:53:35.0931555Z + install_nvidia_driver_common 2023-11-14T04:53:35.0934762Z + echo 'Before installing NVIDIA driver' 2023-11-14T04:53:35.0935529Z + lspci 2023-11-14T04:53:35.0937673Z Before installing NVIDIA driver 2023-11-14T04:53:35.1120938Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2023-11-14T04:53:35.1121960Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2023-11-14T04:53:35.1122971Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2023-11-14T04:53:35.1123907Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2023-11-14T04:53:35.1126738Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2023-11-14T04:53:35.1127673Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2023-11-14T04:53:35.1128729Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-11-14T04:53:35.1129822Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-11-14T04:53:35.1131334Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2023-11-14T04:53:35.1132105Z + lsmod 2023-11-14T04:53:35.1144444Z Module Size Used by 2023-11-14T04:53:35.1144985Z ib_core 266240 0 2023-11-14T04:53:35.1145462Z nvidia_modeset 1306624 0 2023-11-14T04:53:35.1145909Z veth 16384 0 2023-11-14T04:53:35.1146369Z nvidia_uvm 1466368 0 2023-11-14T04:53:35.1146939Z nvidia 56422400 2 nvidia_uvm,nvidia_modeset 2023-11-14T04:53:35.1147531Z drm 425984 1 nvidia 2023-11-14T04:53:35.1148284Z i2c_core 77824 2 nvidia,drm 2023-11-14T04:53:35.1148907Z backlight 16384 1 nvidia_modeset 2023-11-14T04:53:35.1149468Z xt_conntrack 16384 1 2023-11-14T04:53:35.1149949Z ipt_MASQUERADE 16384 1 2023-11-14T04:53:35.1150499Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2023-11-14T04:53:35.1151071Z nf_conntrack_netlink 49152 0 2023-11-14T04:53:35.1151629Z nfnetlink 16384 2 nf_conntrack_netlink 2023-11-14T04:53:35.1152214Z xfrm_user 45056 1 2023-11-14T04:53:35.1152691Z xfrm_algo 16384 1 xfrm_user 2023-11-14T04:53:35.1153230Z iptable_nat 16384 1 2023-11-14T04:53:35.1153709Z nf_conntrack_ipv4 16384 3 2023-11-14T04:53:35.1154238Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2023-11-14T04:53:35.1154948Z nf_nat_ipv4 16384 1 iptable_nat 2023-11-14T04:53:35.1155608Z nf_nat 36864 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2023-11-14T04:53:35.1156815Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2023-11-14T04:53:35.1157874Z xt_addrtype 16384 2 2023-11-14T04:53:35.1158353Z iptable_filter 16384 1 2023-11-14T04:53:35.1158829Z br_netfilter 24576 0 2023-11-14T04:53:35.1159328Z bridge 172032 1 br_netfilter 2023-11-14T04:53:35.1159880Z stp 16384 1 bridge 2023-11-14T04:53:35.1160536Z llc 16384 2 bridge,stp 2023-11-14T04:53:35.1161530Z overlay 86016 0 2023-11-14T04:53:35.1162793Z sunrpc 393216 1 2023-11-14T04:53:35.1163424Z dm_mirror 28672 0 2023-11-14T04:53:35.1163967Z dm_region_hash 20480 1 dm_mirror 2023-11-14T04:53:35.1164721Z dm_log 20480 2 dm_region_hash,dm_mirror 2023-11-14T04:53:35.1165474Z dm_mod 143360 2 dm_log,dm_mirror 2023-11-14T04:53:35.1166199Z dax 69632 1 dm_mod 2023-11-14T04:53:35.1166787Z sb_edac 24576 0 2023-11-14T04:53:35.1167353Z crc32_pclmul 16384 0 2023-11-14T04:53:35.1167913Z ghash_clmulni_intel 16384 0 2023-11-14T04:53:35.1168484Z pcbc 16384 0 2023-11-14T04:53:35.1169036Z ata_piix 36864 0 2023-11-14T04:53:35.1169596Z libata 266240 1 ata_piix 2023-11-14T04:53:35.1170206Z aesni_intel 188416 0 2023-11-14T04:53:35.1207449Z aes_x86_64 20480 1 aesni_intel 2023-11-14T04:53:35.1208122Z crypto_simd 16384 1 aesni_intel 2023-11-14T04:53:35.1208677Z glue_helper 16384 1 aesni_intel 2023-11-14T04:53:35.1209458Z pcc_cpufreq 16384 0 2023-11-14T04:53:35.1210110Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2023-11-14T04:53:35.1211395Z scsi_mod 245760 1 libata 2023-11-14T04:53:35.1211926Z mousedev 24576 0 2023-11-14T04:53:35.1212392Z psmouse 32768 0 2023-11-14T04:53:35.1212832Z evdev 20480 3 2023-11-14T04:53:35.1213295Z button 16384 0 2023-11-14T04:53:35.1213751Z ena 114688 0 2023-11-14T04:53:35.1214189Z xen_blkfront 49152 2 2023-11-14T04:53:35.1214651Z crc32c_intel 24576 0 2023-11-14T04:53:35.1215107Z autofs4 49152 2 2023-11-14T04:53:35.1215536Z + modinfo nvidia 2023-11-14T04:53:35.1216429Z filename: /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2023-11-14T04:53:35.1217307Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2023-11-14T04:53:35.1217928Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2023-11-14T04:53:35.1218530Z alias: char-major-195-* 2023-11-14T04:53:35.1219006Z version: 535.54.03 2023-11-14T04:53:35.1219433Z supported: external 2023-11-14T04:53:35.1219836Z license: NVIDIA 2023-11-14T04:53:35.1220275Z srcversion: EA9C7EF32617E104C8240C4 2023-11-14T04:53:35.1220840Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2023-11-14T04:53:35.1221627Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2023-11-14T04:53:35.1222280Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2023-11-14T04:53:35.1222894Z depends: i2c-core,drm 2023-11-14T04:53:35.1223322Z retpoline: Y 2023-11-14T04:53:35.1223699Z name: nvidia 2023-11-14T04:53:35.1224388Z vermagic: 4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 2023-11-14T04:53:35.1225195Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2023-11-14T04:53:35.1226010Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2023-11-14T04:53:35.1226770Z parm: NVreg_ResmanDebugLevel:int 2023-11-14T04:53:35.1227304Z parm: NVreg_RmLogonRC:int 2023-11-14T04:53:35.1227842Z parm: NVreg_ModifyDeviceFiles:int 2023-11-14T04:53:35.1228412Z parm: NVreg_DeviceFileUID:int 2023-11-14T04:53:35.1228935Z parm: NVreg_DeviceFileGID:int 2023-11-14T04:53:35.1229487Z parm: NVreg_DeviceFileMode:int 2023-11-14T04:53:35.1230152Z parm: NVreg_InitializeSystemMemoryAllocations:int 2023-11-14T04:53:35.1230844Z parm: NVreg_UsePageAttributeTable:int 2023-11-14T04:53:35.1231443Z parm: NVreg_EnablePCIeGen3:int 2023-11-14T04:53:35.1231980Z parm: NVreg_EnableMSI:int 2023-11-14T04:53:35.1232476Z parm: NVreg_TCEBypassMode:int 2023-11-14T04:53:35.1233042Z parm: NVreg_EnableStreamMemOPs:int 2023-11-14T04:53:35.1233702Z parm: NVreg_RestrictProfilingToAdminUsers:int 2023-11-14T04:53:35.1234414Z parm: NVreg_PreserveVideoMemoryAllocations:int 2023-11-14T04:53:35.1235126Z parm: NVreg_EnableS0ixPowerManagement:int 2023-11-14T04:53:35.1235894Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2023-11-14T04:53:35.1236628Z parm: NVreg_DynamicPowerManagement:int 2023-11-14T04:53:35.1237400Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2023-11-14T04:53:35.1238147Z parm: NVreg_EnableGpuFirmware:int 2023-11-14T04:53:35.1238753Z parm: NVreg_EnableGpuFirmwareLogs:int 2023-11-14T04:53:35.1239421Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2023-11-14T04:53:35.1240106Z parm: NVreg_EnableUserNUMAManagement:int 2023-11-14T04:53:35.1240711Z parm: NVreg_MemoryPoolSize:int 2023-11-14T04:53:35.1241287Z parm: NVreg_KMallocHeapMaxSize:int 2023-11-14T04:53:35.1241891Z parm: NVreg_VMallocHeapMaxSize:int 2023-11-14T04:53:35.1242462Z parm: NVreg_IgnoreMMIOCheck:int 2023-11-14T04:53:35.1243017Z parm: NVreg_NvLinkDisable:int 2023-11-14T04:53:35.1243826Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2023-11-14T04:53:35.1244485Z parm: NVreg_RegisterPCIDriver:int 2023-11-14T04:53:35.1245063Z parm: NVreg_EnableResizableBar:int 2023-11-14T04:53:35.1245670Z parm: NVreg_EnableDbgBreakpoint:int 2023-11-14T04:53:35.1246269Z parm: NVreg_RegistryDwords:charp 2023-11-14T04:53:35.1246879Z parm: NVreg_RegistryDwordsPerDevice:charp 2023-11-14T04:53:35.1247481Z parm: NVreg_RmMsg:charp 2023-11-14T04:53:35.1247996Z parm: NVreg_GpuBlacklist:charp 2023-11-14T04:53:35.1248561Z parm: NVreg_TemporaryFilePath:charp 2023-11-14T04:53:35.1249146Z parm: NVreg_ExcludedGpus:charp 2023-11-14T04:53:35.1249711Z parm: NVreg_DmaRemapPeerMmio:int 2023-11-14T04:53:35.1250291Z parm: NVreg_RmNvlinkBandwidth:charp 2023-11-14T04:53:35.1251540Z parm: rm_firmware_active:charp 2023-11-14T04:53:35.1252071Z + HAS_NVIDIA_DRIVER=0 2023-11-14T04:53:35.1252539Z ++ command -v nvidia-smi 2023-11-14T04:53:35.1253029Z + '[' -x /usr/bin/nvidia-smi ']' 2023-11-14T04:53:35.1253487Z + set +e 2023-11-14T04:53:35.1254099Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2023-11-14T04:53:37.3961626Z + INSTALLED_DRIVER_VERSION=535.54.03 2023-11-14T04:53:37.3962743Z + NVIDIA_SMI_STATUS=0 2023-11-14T04:53:37.3963712Z + '[' 0 -ne 0 ']' 2023-11-14T04:53:37.3964767Z + '[' 535.54.03 '!=' 535.54.03 ']' 2023-11-14T04:53:37.3965288Z + HAS_NVIDIA_DRIVER=1 2023-11-14T04:53:37.3966206Z + echo 'NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation' 2023-11-14T04:53:37.3967087Z + set -e 2023-11-14T04:53:37.3967460Z + '[' 1 -eq 0 ']' 2023-11-14T04:53:37.3968183Z NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation 2023-11-14T04:53:37.3969049Z + post_install_nvidia_driver_common 2023-11-14T04:53:37.3969553Z + sudo modprobe nvidia 2023-11-14T04:53:37.4111771Z + echo 'After installing NVIDIA driver' 2023-11-14T04:53:37.4112307Z + lspci 2023-11-14T04:53:37.4114626Z After installing NVIDIA driver 2023-11-14T04:53:37.4282407Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2023-11-14T04:53:37.4283422Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2023-11-14T04:53:37.4284395Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2023-11-14T04:53:37.4285359Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2023-11-14T04:53:37.4286178Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2023-11-14T04:53:37.4287020Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2023-11-14T04:53:37.4288061Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-11-14T04:53:37.4289140Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-11-14T04:53:37.4290166Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2023-11-14T04:53:37.4291530Z + lsmod 2023-11-14T04:53:37.4305598Z Module Size Used by 2023-11-14T04:53:37.4306131Z ib_core 266240 0 2023-11-14T04:53:37.4306588Z nvidia_modeset 1306624 0 2023-11-14T04:53:37.4307057Z veth 16384 0 2023-11-14T04:53:37.4307519Z nvidia_uvm 1466368 0 2023-11-14T04:53:37.4308069Z nvidia 56422400 2 nvidia_uvm,nvidia_modeset 2023-11-14T04:53:37.4308706Z drm 425984 1 nvidia 2023-11-14T04:53:37.4309231Z i2c_core 77824 2 nvidia,drm 2023-11-14T04:53:37.4309797Z backlight 16384 1 nvidia_modeset 2023-11-14T04:53:37.4310354Z xt_conntrack 16384 1 2023-11-14T04:53:37.4310831Z ipt_MASQUERADE 16384 1 2023-11-14T04:53:37.4311409Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2023-11-14T04:53:37.4312003Z nf_conntrack_netlink 49152 0 2023-11-14T04:53:37.4312561Z nfnetlink 16384 2 nf_conntrack_netlink 2023-11-14T04:53:37.4313379Z xfrm_user 45056 1 2023-11-14T04:53:37.4313870Z xfrm_algo 16384 1 xfrm_user 2023-11-14T04:53:37.4314392Z iptable_nat 16384 1 2023-11-14T04:53:37.4314849Z nf_conntrack_ipv4 16384 3 2023-11-14T04:53:37.4315390Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2023-11-14T04:53:37.4315993Z nf_nat_ipv4 16384 1 iptable_nat 2023-11-14T04:53:37.4316659Z nf_nat 36864 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2023-11-14T04:53:37.4317841Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2023-11-14T04:53:37.4318916Z xt_addrtype 16384 2 2023-11-14T04:53:37.4319395Z iptable_filter 16384 1 2023-11-14T04:53:37.4319854Z br_netfilter 24576 0 2023-11-14T04:53:37.4320366Z bridge 172032 1 br_netfilter 2023-11-14T04:53:37.4320917Z stp 16384 1 bridge 2023-11-14T04:53:37.4321424Z llc 16384 2 bridge,stp 2023-11-14T04:53:37.4321948Z overlay 86016 0 2023-11-14T04:53:37.4322410Z sunrpc 393216 1 2023-11-14T04:53:37.4322856Z dm_mirror 28672 0 2023-11-14T04:53:37.4323352Z dm_region_hash 20480 1 dm_mirror 2023-11-14T04:53:37.4323966Z dm_log 20480 2 dm_region_hash,dm_mirror 2023-11-14T04:53:37.4324717Z dm_mod 143360 2 dm_log,dm_mirror 2023-11-14T04:53:37.4325303Z dax 69632 1 dm_mod 2023-11-14T04:53:37.4325803Z sb_edac 24576 0 2023-11-14T04:53:37.4326249Z crc32_pclmul 16384 0 2023-11-14T04:53:37.4326724Z ghash_clmulni_intel 16384 0 2023-11-14T04:53:37.4327201Z pcbc 16384 0 2023-11-14T04:53:37.4327641Z ata_piix 36864 0 2023-11-14T04:53:37.4328125Z libata 266240 1 ata_piix 2023-11-14T04:53:37.4328638Z aesni_intel 188416 0 2023-11-14T04:53:37.4329114Z aes_x86_64 20480 1 aesni_intel 2023-11-14T04:53:37.4329682Z crypto_simd 16384 1 aesni_intel 2023-11-14T04:53:37.4330251Z glue_helper 16384 1 aesni_intel 2023-11-14T04:53:37.4331221Z pcc_cpufreq 16384 0 2023-11-14T04:53:37.4331952Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2023-11-14T04:53:37.4332646Z scsi_mod 245760 1 libata 2023-11-14T04:53:37.4333286Z mousedev 24576 0 2023-11-14T04:53:37.4333756Z psmouse 32768 0 2023-11-14T04:53:37.4334219Z evdev 20480 3 2023-11-14T04:53:37.4334657Z button 16384 0 2023-11-14T04:53:37.4335117Z ena 114688 0 2023-11-14T04:53:37.4335578Z xen_blkfront 49152 2 2023-11-14T04:53:37.4336022Z crc32c_intel 24576 0 2023-11-14T04:53:37.4336481Z autofs4 49152 2 2023-11-14T04:53:37.4336930Z + modinfo nvidia 2023-11-14T04:53:37.4337793Z filename: /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2023-11-14T04:53:37.4338661Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2023-11-14T04:53:37.4339264Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2023-11-14T04:53:37.4339880Z alias: char-major-195-* 2023-11-14T04:53:37.4340338Z version: 535.54.03 2023-11-14T04:53:37.4340770Z supported: external 2023-11-14T04:53:37.4341195Z license: NVIDIA 2023-11-14T04:53:37.4341624Z srcversion: EA9C7EF32617E104C8240C4 2023-11-14T04:53:37.4342214Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2023-11-14T04:53:37.4342838Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2023-11-14T04:53:37.4343434Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2023-11-14T04:53:37.4344050Z depends: i2c-core,drm 2023-11-14T04:53:37.4344500Z retpoline: Y 2023-11-14T04:53:37.4344864Z name: nvidia 2023-11-14T04:53:37.4345565Z vermagic: 4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 2023-11-14T04:53:37.4346397Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2023-11-14T04:53:37.4347384Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2023-11-14T04:53:37.4348131Z parm: NVreg_ResmanDebugLevel:int 2023-11-14T04:53:37.4348690Z parm: NVreg_RmLogonRC:int 2023-11-14T04:53:37.4349215Z parm: NVreg_ModifyDeviceFiles:int 2023-11-14T04:53:37.4349784Z parm: NVreg_DeviceFileUID:int 2023-11-14T04:53:37.4350335Z parm: NVreg_DeviceFileGID:int 2023-11-14T04:53:37.4350868Z parm: NVreg_DeviceFileMode:int 2023-11-14T04:53:37.4351533Z parm: NVreg_InitializeSystemMemoryAllocations:int 2023-11-14T04:53:37.4352242Z parm: NVreg_UsePageAttributeTable:int 2023-11-14T04:53:37.4352826Z parm: NVreg_EnablePCIeGen3:int 2023-11-14T04:53:37.4353364Z parm: NVreg_EnableMSI:int 2023-11-14T04:53:37.4353886Z parm: NVreg_TCEBypassMode:int 2023-11-14T04:53:37.4354444Z parm: NVreg_EnableStreamMemOPs:int 2023-11-14T04:53:37.4355116Z parm: NVreg_RestrictProfilingToAdminUsers:int 2023-11-14T04:53:37.4355850Z parm: NVreg_PreserveVideoMemoryAllocations:int 2023-11-14T04:53:37.4356538Z parm: NVreg_EnableS0ixPowerManagement:int 2023-11-14T04:53:37.4357309Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2023-11-14T04:53:37.4358067Z parm: NVreg_DynamicPowerManagement:int 2023-11-14T04:53:37.4358943Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2023-11-14T04:53:37.4359721Z parm: NVreg_EnableGpuFirmware:int 2023-11-14T04:53:37.4360336Z parm: NVreg_EnableGpuFirmwareLogs:int 2023-11-14T04:53:37.4361009Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2023-11-14T04:53:37.4361676Z parm: NVreg_EnableUserNUMAManagement:int 2023-11-14T04:53:37.4362300Z parm: NVreg_MemoryPoolSize:int 2023-11-14T04:53:37.4362882Z parm: NVreg_KMallocHeapMaxSize:int 2023-11-14T04:53:37.4363471Z parm: NVreg_VMallocHeapMaxSize:int 2023-11-14T04:53:37.4364066Z parm: NVreg_IgnoreMMIOCheck:int 2023-11-14T04:53:37.4364627Z parm: NVreg_NvLinkDisable:int 2023-11-14T04:53:37.4365240Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2023-11-14T04:53:37.4365903Z parm: NVreg_RegisterPCIDriver:int 2023-11-14T04:53:37.4366507Z parm: NVreg_EnableResizableBar:int 2023-11-14T04:53:37.4367100Z parm: NVreg_EnableDbgBreakpoint:int 2023-11-14T04:53:37.4367704Z parm: NVreg_RegistryDwords:charp 2023-11-14T04:53:37.4368333Z parm: NVreg_RegistryDwordsPerDevice:charp 2023-11-14T04:53:37.4368914Z parm: NVreg_RmMsg:charp 2023-11-14T04:53:37.4369430Z parm: NVreg_GpuBlacklist:charp 2023-11-14T04:53:37.4370013Z parm: NVreg_TemporaryFilePath:charp 2023-11-14T04:53:37.4370583Z parm: NVreg_ExcludedGpus:charp 2023-11-14T04:53:37.4371613Z parm: NVreg_DmaRemapPeerMmio:int 2023-11-14T04:53:37.4372223Z parm: NVreg_RmNvlinkBandwidth:charp 2023-11-14T04:53:37.4372800Z parm: rm_firmware_active:charp 2023-11-14T04:53:37.4373304Z + set +e 2023-11-14T04:53:37.4373683Z + nvidia-smi 2023-11-14T04:53:38.8272072Z Tue Nov 14 04:53:38 2023 2023-11-14T04:53:38.8272996Z +---------------------------------------------------------------------------------------+ 2023-11-14T04:53:38.8274046Z | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2023-11-14T04:53:38.8275042Z |-----------------------------------------+----------------------+----------------------+ 2023-11-14T04:53:38.8276051Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2023-11-14T04:53:38.8277168Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2023-11-14T04:53:38.8278021Z | | | MIG M. | 2023-11-14T04:53:38.8278717Z |=========================================+======================+======================| 2023-11-14T04:53:38.8319114Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2023-11-14T04:53:38.8320026Z | N/A 25C P0 39W / 150W | 0MiB / 7680MiB | 96% Default | 2023-11-14T04:53:38.8320805Z | | | N/A | 2023-11-14T04:53:38.8321671Z +-----------------------------------------+----------------------+----------------------+ 2023-11-14T04:53:38.8366166Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2023-11-14T04:53:38.8367074Z | N/A 30C P0 38W / 150W | 0MiB / 7680MiB | 6% Default | 2023-11-14T04:53:38.8367842Z | | | N/A | 2023-11-14T04:53:38.8368683Z +-----------------------------------------+----------------------+----------------------+ 2023-11-14T04:53:38.8369407Z 2023-11-14T04:53:38.8370211Z +---------------------------------------------------------------------------------------+ 2023-11-14T04:53:38.8371627Z | Processes: | 2023-11-14T04:53:38.8372520Z | GPU GI CI PID Type Process name GPU Memory | 2023-11-14T04:53:38.8373616Z | ID ID Usage | 2023-11-14T04:53:38.8374356Z |=======================================================================================| 2023-11-14T04:53:38.8375188Z | No running processes found | 2023-11-14T04:53:38.8376160Z +---------------------------------------------------------------------------------------+ 2023-11-14T04:53:39.7803348Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2023-11-14T04:53:41.1675960Z Tesla M60 2023-11-14T04:53:42.0522961Z + NVIDIA_SMI_STATUS=0 2023-11-14T04:53:42.0524167Z + '[' 0 -eq 0 ']' 2023-11-14T04:53:42.0524685Z + echo 'INFO: Ignoring allowed status 0' 2023-11-14T04:53:42.0525206Z + set -e 2023-11-14T04:53:42.0525572Z INFO: Ignoring allowed status 0 2023-11-14T04:53:42.0529579Z == Installing nvidia container toolkit for amzn2 == 2023-11-14T04:53:42.0533874Z + sudo yum install -y yum-utils 2023-11-14T04:53:42.5883163Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-11-14T04:53:44.5740274Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2023-11-14T04:53:44.5741160Z Nothing to do 2023-11-14T04:53:44.7295648Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2023-11-14T04:53:45.2839502Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-11-14T04:53:45.3129287Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2023-11-14T04:53:45.3131343Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2023-11-14T04:53:45.3132590Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2023-11-14T04:53:45.3275380Z + sudo yum install -y nvidia-docker2 2023-11-14T04:53:45.8732985Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-11-14T04:53:47.4593055Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2023-11-14T04:53:47.4593922Z Nothing to do 2023-11-14T04:53:47.6069472Z + sudo systemctl restart docker 2023-11-14T04:54:05.4718323Z Command completed after 1 attempt(s). 2023-11-14T04:54:05.4807127Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2023-11-14T04:54:05.4808213Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2023-11-14T04:54:05.4809114Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2023-11-14T04:54:05.4809968Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:54:05.4820996Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:05.4821636Z env: 2023-11-14T04:54:05.4822008Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:05.4822610Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:05.4823220Z ##[endgroup] 2023-11-14T04:54:05.7803329Z Defaulting to user installation because normal site-packages is not writeable 2023-11-14T04:54:05.8017042Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2023-11-14T04:54:05.8141121Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2023-11-14T04:54:05.9462680Z Prepare all required actions 2023-11-14T04:54:05.9463365Z Getting action download info 2023-11-14T04:54:06.0683147Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2023-11-14T04:54:06.3290369Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2023-11-14T04:54:06.4715512Z ##[group]Run ./.github/actions/download-build-artifacts 2023-11-14T04:54:06.4716099Z with: 2023-11-14T04:54:06.4716455Z name: linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:54:06.4716952Z env: 2023-11-14T04:54:06.4717265Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:06.4717796Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:06.4718381Z ##[endgroup] 2023-11-14T04:54:06.4763393Z ##[group]Run seemethere/download-artifact-s3@v4 2023-11-14T04:54:06.4763925Z with: 2023-11-14T04:54:06.4764291Z name: linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:54:06.4764845Z s3-bucket: gha-artifacts 2023-11-14T04:54:06.4765260Z region: us-east-1 2023-11-14T04:54:06.4765598Z env: 2023-11-14T04:54:06.4765897Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:06.4766432Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:06.4767006Z ##[endgroup] 2023-11-14T04:54:07.0957318Z (node:85450) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2023-11-14T04:54:07.0958711Z 2023-11-14T04:54:07.0959248Z Please migrate your code to use AWS SDK for JavaScript (v3). 2023-11-14T04:54:07.0960621Z For more information, check the migration guide at https://a.co/7PzMCcy 2023-11-14T04:54:07.0962263Z (Use `node --trace-warnings ...` to show where the warning was created) 2023-11-14T04:54:07.2191906Z Found 1 objects with prefix pytorch/pytorch/6859165800/linux-focal-cuda11.8-py3.10-gcc9/ 2023-11-14T04:54:07.2193259Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2023-11-14T04:54:26.4395404Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2023-11-14T04:54:26.4405963Z Artifact download has finished successfully 2023-11-14T04:54:26.4641172Z ##[group]Run unzip -o artifacts.zip 2023-11-14T04:54:26.4641723Z unzip -o artifacts.zip 2023-11-14T04:54:26.4650607Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:26.4651943Z env: 2023-11-14T04:54:26.4652303Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:26.4652869Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:26.4653472Z ##[endgroup] 2023-11-14T04:54:26.4696222Z Archive: artifacts.zip 2023-11-14T04:54:26.4698473Z creating: dist/ 2023-11-14T04:54:28.7859984Z inflating: dist/torch-2.2.0a0+gitfe705c0-cp310-cp310-linux_x86_64.whl 2023-11-14T04:54:28.7860850Z creating: build/custom_test_artifacts/ 2023-11-14T04:54:28.7863245Z creating: build/custom_test_artifacts/custom-op-build/ 2023-11-14T04:54:28.7864245Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2023-11-14T04:54:28.7865298Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2023-11-14T04:54:28.7870269Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2023-11-14T04:54:28.7871493Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2023-11-14T04:54:28.7872996Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2023-11-14T04:54:28.7874274Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2023-11-14T04:54:28.7875515Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2023-11-14T04:54:28.7876943Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2023-11-14T04:54:28.7878364Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2023-11-14T04:54:28.7879924Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2023-11-14T04:54:28.7881214Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2023-11-14T04:54:28.7882706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-11-14T04:54:28.7884236Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2023-11-14T04:54:28.7885713Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2023-11-14T04:54:28.7887163Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2023-11-14T04:54:28.7888671Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2023-11-14T04:54:28.7890180Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2023-11-14T04:54:28.7891891Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2023-11-14T04:54:28.7893170Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2023-11-14T04:54:28.7948917Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-11-14T04:54:28.7950777Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-11-14T04:54:28.7952642Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-11-14T04:54:28.7954576Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-11-14T04:54:28.7956447Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-11-14T04:54:28.7958242Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-11-14T04:54:28.7960049Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-11-14T04:54:28.7961866Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-11-14T04:54:28.7963649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-11-14T04:54:28.8005631Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-11-14T04:54:28.8056137Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-11-14T04:54:28.8057951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-11-14T04:54:28.8059598Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-11-14T04:54:28.8061204Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-11-14T04:54:28.8063000Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-11-14T04:54:28.8064583Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-11-14T04:54:28.8066129Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2023-11-14T04:54:28.8067696Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-11-14T04:54:28.8140155Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2023-11-14T04:54:28.8216277Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2023-11-14T04:54:28.8217897Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2023-11-14T04:54:28.8219172Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2023-11-14T04:54:28.8220270Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2023-11-14T04:54:28.8221437Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2023-11-14T04:54:28.8222620Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2023-11-14T04:54:28.8223930Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2023-11-14T04:54:28.8225413Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2023-11-14T04:54:28.8226821Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2023-11-14T04:54:28.8228152Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2023-11-14T04:54:28.8229551Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2023-11-14T04:54:28.8230934Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2023-11-14T04:54:28.8232315Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2023-11-14T04:54:28.8233688Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2023-11-14T04:54:28.8235050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2023-11-14T04:54:28.8250661Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2023-11-14T04:54:28.8381030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2023-11-14T04:54:28.8382324Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2023-11-14T04:54:28.8383714Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2023-11-14T04:54:28.8385294Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2023-11-14T04:54:28.8386791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2023-11-14T04:54:28.8388195Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2023-11-14T04:54:28.8389649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2023-11-14T04:54:28.8391096Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2023-11-14T04:54:28.8392567Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2023-11-14T04:54:28.8394026Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2023-11-14T04:54:28.8395459Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2023-11-14T04:54:28.8414655Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2023-11-14T04:54:28.8515576Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2023-11-14T04:54:28.8517163Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-11-14T04:54:28.8518563Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2023-11-14T04:54:28.8519813Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2023-11-14T04:54:28.8521154Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2023-11-14T04:54:28.8522328Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2023-11-14T04:54:28.8523485Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2023-11-14T04:54:28.8524588Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2023-11-14T04:54:28.8525565Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2023-11-14T04:54:28.8526594Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2023-11-14T04:54:28.8633719Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2023-11-14T04:54:28.8709392Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2023-11-14T04:54:28.8710317Z creating: build/custom_test_artifacts/jit-hook-build/ 2023-11-14T04:54:28.8711194Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2023-11-14T04:54:28.8712240Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2023-11-14T04:54:28.8719334Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2023-11-14T04:54:28.8720519Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2023-11-14T04:54:28.8721714Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2023-11-14T04:54:28.8722964Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2023-11-14T04:54:28.8724175Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2023-11-14T04:54:28.8725574Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2023-11-14T04:54:28.8726997Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2023-11-14T04:54:28.8728285Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2023-11-14T04:54:28.8729570Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2023-11-14T04:54:28.8731766Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-11-14T04:54:28.8733312Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2023-11-14T04:54:28.8734953Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2023-11-14T04:54:28.8736419Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2023-11-14T04:54:28.8737902Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2023-11-14T04:54:28.8739376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2023-11-14T04:54:28.8740679Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2023-11-14T04:54:28.8741967Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2023-11-14T04:54:28.8797409Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-11-14T04:54:28.8799479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-11-14T04:54:28.8801332Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-11-14T04:54:28.8803212Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-11-14T04:54:28.8805097Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-11-14T04:54:28.8807005Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-11-14T04:54:28.8808825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-11-14T04:54:28.8810618Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-11-14T04:54:28.8812720Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-11-14T04:54:28.8854610Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-11-14T04:54:28.8904525Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-11-14T04:54:28.8906556Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-11-14T04:54:28.8908243Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-11-14T04:54:28.8909831Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-11-14T04:54:28.8911371Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-11-14T04:54:28.8912990Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-11-14T04:54:28.8914501Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2023-11-14T04:54:28.8916063Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-11-14T04:54:28.8988534Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2023-11-14T04:54:28.9064686Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2023-11-14T04:54:28.9066192Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2023-11-14T04:54:28.9067459Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2023-11-14T04:54:28.9068560Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2023-11-14T04:54:28.9069693Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2023-11-14T04:54:28.9070898Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2023-11-14T04:54:28.9072271Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2023-11-14T04:54:28.9073768Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2023-11-14T04:54:28.9075230Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2023-11-14T04:54:28.9076609Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2023-11-14T04:54:28.9078042Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2023-11-14T04:54:28.9079452Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2023-11-14T04:54:28.9081102Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2023-11-14T04:54:28.9082526Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2023-11-14T04:54:28.9083935Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2023-11-14T04:54:28.9099676Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2023-11-14T04:54:28.9178620Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2023-11-14T04:54:28.9180193Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-11-14T04:54:28.9181570Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2023-11-14T04:54:28.9182792Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2023-11-14T04:54:28.9183963Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2023-11-14T04:54:28.9207591Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2023-11-14T04:54:28.9208871Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2023-11-14T04:54:28.9209974Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2023-11-14T04:54:28.9211248Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2023-11-14T04:54:28.9212276Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2023-11-14T04:54:28.9247511Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2023-11-14T04:54:28.9248483Z creating: build/custom_test_artifacts/custom-backend-build/ 2023-11-14T04:54:28.9249427Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2023-11-14T04:54:28.9250569Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2023-11-14T04:54:28.9257591Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2023-11-14T04:54:28.9258866Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2023-11-14T04:54:28.9260129Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2023-11-14T04:54:28.9261479Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2023-11-14T04:54:28.9262820Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2023-11-14T04:54:28.9264317Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2023-11-14T04:54:28.9265811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2023-11-14T04:54:28.9267176Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2023-11-14T04:54:28.9268522Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2023-11-14T04:54:28.9270066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-11-14T04:54:28.9271639Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2023-11-14T04:54:28.9273184Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2023-11-14T04:54:28.9274743Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2023-11-14T04:54:28.9276290Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2023-11-14T04:54:28.9277876Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2023-11-14T04:54:28.9279525Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2023-11-14T04:54:28.9280891Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2023-11-14T04:54:28.9335348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-11-14T04:54:28.9337287Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-11-14T04:54:28.9339422Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-11-14T04:54:28.9341481Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-11-14T04:54:28.9343432Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-11-14T04:54:28.9345319Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-11-14T04:54:28.9347203Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-11-14T04:54:28.9349097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-11-14T04:54:28.9350960Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-11-14T04:54:28.9392347Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-11-14T04:54:28.9442172Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-11-14T04:54:28.9444069Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-11-14T04:54:28.9445793Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-11-14T04:54:28.9447727Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-11-14T04:54:28.9449385Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-11-14T04:54:28.9451366Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-11-14T04:54:28.9452984Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2023-11-14T04:54:28.9454641Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-11-14T04:54:28.9525867Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2023-11-14T04:54:28.9602205Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2023-11-14T04:54:28.9603800Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2023-11-14T04:54:28.9605149Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2023-11-14T04:54:28.9606298Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2023-11-14T04:54:28.9607528Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2023-11-14T04:54:28.9608802Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2023-11-14T04:54:28.9610237Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2023-11-14T04:54:28.9612347Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2023-11-14T04:54:28.9613888Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2023-11-14T04:54:28.9615326Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2023-11-14T04:54:28.9616873Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2023-11-14T04:54:28.9618371Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2023-11-14T04:54:28.9620026Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2023-11-14T04:54:28.9621587Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2023-11-14T04:54:28.9623099Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2023-11-14T04:54:28.9624715Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2023-11-14T04:54:28.9769891Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2023-11-14T04:54:28.9771702Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2023-11-14T04:54:28.9773237Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2023-11-14T04:54:28.9774942Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2023-11-14T04:54:28.9776562Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2023-11-14T04:54:28.9778121Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2023-11-14T04:54:28.9779730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2023-11-14T04:54:28.9781311Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2023-11-14T04:54:28.9782919Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2023-11-14T04:54:28.9784517Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2023-11-14T04:54:28.9786097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2023-11-14T04:54:28.9802275Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2023-11-14T04:54:28.9869027Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2023-11-14T04:54:28.9870738Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-11-14T04:54:28.9872211Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2023-11-14T04:54:28.9873514Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2023-11-14T04:54:28.9874753Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2023-11-14T04:54:28.9875971Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2023-11-14T04:54:28.9877204Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2023-11-14T04:54:28.9878358Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2023-11-14T04:54:28.9879418Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2023-11-14T04:54:28.9880512Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2023-11-14T04:54:29.0003426Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2023-11-14T04:54:29.0055354Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2023-11-14T04:54:29.0056161Z creating: build/lib/ 2023-11-14T04:54:29.0056612Z inflating: build/lib/libclog.a 2023-11-14T04:54:29.0067016Z inflating: build/lib/libpthreadpool.a 2023-11-14T04:54:29.0148237Z inflating: build/lib/libgtest.a 2023-11-14T04:54:29.0260360Z inflating: build/lib/libprotobuf-lite.a 2023-11-14T04:54:29.0353635Z inflating: build/lib/libbenchmark.a 2023-11-14T04:54:29.0363293Z inflating: build/lib/libittnotify.a 2023-11-14T04:54:29.0397910Z inflating: build/lib/libtensorpipe_uv.a 2023-11-14T04:54:29.0967893Z inflating: build/lib/libprotobuf.a 2023-11-14T04:54:29.1047879Z inflating: build/lib/libasmjit.a 2023-11-14T04:54:29.1206811Z inflating: build/lib/libgloo.a 2023-11-14T04:54:29.1233692Z inflating: build/lib/libfmt.a 2023-11-14T04:54:29.1234277Z inflating: build/lib/libfoxi_loader.a 2023-11-14T04:54:29.1236662Z inflating: build/lib/libcaffe2_nvrtc.so 2023-11-14T04:54:29.1238055Z inflating: build/lib/libtorch_global_deps.so 2023-11-14T04:54:29.1248420Z inflating: build/lib/libcpuinfo.a 2023-11-14T04:54:29.1258479Z inflating: build/lib/libcpuinfo_internals.a 2023-11-14T04:54:29.1887003Z inflating: build/lib/libprotoc.a 2023-11-14T04:54:29.1903934Z inflating: build/lib/libqnnpack.a 2023-11-14T04:54:29.1928237Z inflating: build/lib/libpytorch_qnnpack.a 2023-11-14T04:54:29.1930947Z inflating: build/lib/libnnpack_reference_layers.a 2023-11-14T04:54:29.1953653Z inflating: build/lib/libnnpack.a 2023-11-14T04:54:29.1976373Z inflating: build/lib/libgmock.a 2023-11-14T04:54:29.1976935Z inflating: build/lib/libgtest_main.a 2023-11-14T04:54:29.1977882Z inflating: build/lib/libbenchmark_main.a 2023-11-14T04:54:29.2694468Z inflating: build/lib/libtensorpipe.a 2023-11-14T04:54:30.3856435Z inflating: build/lib/libdnnl.a 2023-11-14T04:54:30.3964347Z inflating: build/lib/libc10.so 2023-11-14T04:54:30.3964922Z inflating: build/lib/libgmock_main.a 2023-11-14T04:54:30.5529422Z inflating: build/lib/libfbgemm.a 2023-11-14T04:54:30.6151943Z inflating: build/lib/libkineto.a 2023-11-14T04:54:30.6471888Z inflating: build/lib/libtensorpipe_cuda.a 2023-11-14T04:54:30.6520260Z inflating: build/lib/libcaffe2_protos.a 2023-11-14T04:54:30.6586601Z inflating: build/lib/libc10_cuda.so 2023-11-14T04:54:30.6638715Z inflating: build/lib/libonnx_proto.a 2023-11-14T04:54:30.6783890Z inflating: build/lib/libXNNPACK.a 2023-11-14T04:54:30.7270137Z inflating: build/lib/libgloo_cuda.a 2023-11-14T04:54:30.8115792Z inflating: build/lib/libonnx.a 2023-11-14T04:54:33.6398222Z inflating: build/lib/libtorch_cpu.so 2023-11-14T04:54:33.6403644Z inflating: build/lib/libunbox_lib.a 2023-11-14T04:54:35.8972977Z inflating: build/lib/libtorch_cuda.so 2023-11-14T04:54:35.8973978Z inflating: build/lib/libtorch.so 2023-11-14T04:54:35.8977578Z inflating: build/lib/libc10d_cuda_test.so 2023-11-14T04:54:37.0393017Z inflating: build/lib/libtorch_cuda_linalg.so 2023-11-14T04:54:37.0418249Z inflating: build/lib/libjitbackend_test.so 2023-11-14T04:54:37.0483118Z inflating: build/lib/libtorchbind_test.so 2023-11-14T04:54:37.0514072Z inflating: build/lib/libbackend_with_compiler.so 2023-11-14T04:54:37.0519189Z inflating: build/lib/libshm.so 2023-11-14T04:54:37.2744622Z inflating: build/lib/libtorch_python.so 2023-11-14T04:54:37.2787243Z inflating: build/lib/libnnapi_backend.so 2023-11-14T04:54:37.2787828Z creating: build/bin/ 2023-11-14T04:54:37.3346797Z inflating: build/bin/protoc-3.13.0.0 2023-11-14T04:54:37.3905618Z inflating: build/bin/protoc 2023-11-14T04:54:37.3968536Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2023-11-14T04:54:37.4033182Z inflating: build/bin/c10_DeviceGuard_test 2023-11-14T04:54:37.4096220Z inflating: build/bin/c10_Device_test 2023-11-14T04:54:37.4170189Z inflating: build/bin/c10_DispatchKeySet_test 2023-11-14T04:54:37.4231909Z inflating: build/bin/c10_StreamGuard_test 2023-11-14T04:54:37.4298870Z inflating: build/bin/c10_SymInt_test 2023-11-14T04:54:37.4367194Z inflating: build/bin/c10_InlineDeviceGuard_test 2023-11-14T04:54:37.4437195Z inflating: build/bin/c10_InlineStreamGuard_test 2023-11-14T04:54:37.4507995Z inflating: build/bin/c10_SizesAndStrides_test 2023-11-14T04:54:37.4596374Z inflating: build/bin/c10_cow_test 2023-11-14T04:54:37.4663270Z inflating: build/bin/c10_Bitset_test 2023-11-14T04:54:37.4724392Z inflating: build/bin/c10_ConstexprCrc_test 2023-11-14T04:54:37.4787126Z inflating: build/bin/c10_DeadlockDetection_test 2023-11-14T04:54:37.4849229Z inflating: build/bin/c10_Half_test 2023-11-14T04:54:37.4919818Z inflating: build/bin/c10_LeftRight_test 2023-11-14T04:54:37.4988837Z inflating: build/bin/c10_Metaprogramming_test 2023-11-14T04:54:37.5051343Z inflating: build/bin/c10_Synchronized_test 2023-11-14T04:54:37.5121401Z inflating: build/bin/c10_ThreadLocal_test 2023-11-14T04:54:37.5186182Z inflating: build/bin/c10_TypeIndex_test 2023-11-14T04:54:37.5249871Z inflating: build/bin/c10_TypeList_test 2023-11-14T04:54:37.5311590Z inflating: build/bin/c10_TypeTraits_test 2023-11-14T04:54:37.5376541Z inflating: build/bin/c10_accumulate_test 2023-11-14T04:54:37.5446102Z inflating: build/bin/c10_bfloat16_test 2023-11-14T04:54:37.5509579Z inflating: build/bin/c10_bit_cast_test 2023-11-14T04:54:37.5580404Z inflating: build/bin/c10_complex_math_test 2023-11-14T04:54:37.5650244Z inflating: build/bin/c10_complex_test 2023-11-14T04:54:37.5788699Z inflating: build/bin/c10_either_test 2023-11-14T04:54:37.5854727Z inflating: build/bin/c10_exception_test 2023-11-14T04:54:37.5917833Z inflating: build/bin/c10_flags_test 2023-11-14T04:54:37.5981622Z inflating: build/bin/c10_irange_test 2023-11-14T04:54:37.6188836Z inflating: build/bin/c10_intrusive_ptr_test 2023-11-14T04:54:37.6259037Z inflating: build/bin/c10_logging_test 2023-11-14T04:54:37.6352824Z inflating: build/bin/c10_optional_test 2023-11-14T04:54:37.6420512Z inflating: build/bin/c10_registry_test 2023-11-14T04:54:37.6499088Z inflating: build/bin/c10_ordered_preserving_dict_test 2023-11-14T04:54:37.6688685Z inflating: build/bin/c10_small_vector_test 2023-11-14T04:54:37.6753353Z inflating: build/bin/c10_ssize_test 2023-11-14T04:54:37.6816691Z inflating: build/bin/c10_tempfile_test 2023-11-14T04:54:37.6890068Z inflating: build/bin/c10_string_view_test 2023-11-14T04:54:37.6960569Z inflating: build/bin/c10_typeid_test 2023-11-14T04:54:37.7022322Z inflating: build/bin/c10_intrusive_ptr_benchmark 2023-11-14T04:54:37.7088490Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2023-11-14T04:54:37.7155052Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2023-11-14T04:54:37.7221481Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2023-11-14T04:54:37.7286692Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2023-11-14T04:54:37.7348400Z inflating: build/bin/c10_cuda_CUDATest 2023-11-14T04:54:37.7414891Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2023-11-14T04:54:37.7481059Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2023-11-14T04:54:37.7547278Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2023-11-14T04:54:37.7922002Z inflating: build/bin/vec_test_all_types_DEFAULT 2023-11-14T04:54:37.8316692Z inflating: build/bin/vec_test_all_types_AVX512 2023-11-14T04:54:37.8730495Z inflating: build/bin/vec_test_all_types_AVX2 2023-11-14T04:54:37.8797514Z inflating: build/bin/HashStoreTest 2023-11-14T04:54:37.8866580Z inflating: build/bin/TCPStoreTest 2023-11-14T04:54:37.8932983Z inflating: build/bin/FileStoreTest 2023-11-14T04:54:37.8949921Z inflating: build/bin/ProcessGroupMPITest 2023-11-14T04:54:37.9017922Z inflating: build/bin/test_edge_op_registration 2023-11-14T04:54:37.9021322Z inflating: build/bin/example_allreduce 2023-11-14T04:54:37.9086240Z inflating: build/bin/Dimname_test 2023-11-14T04:54:37.9178527Z inflating: build/bin/Dict_test 2023-11-14T04:54:37.9259317Z inflating: build/bin/MaybeOwned_test 2023-11-14T04:54:37.9330888Z inflating: build/bin/NamedTensor_test 2023-11-14T04:54:37.9404925Z inflating: build/bin/apply_utils_test 2023-11-14T04:54:37.9478314Z inflating: build/bin/atest 2023-11-14T04:54:37.9547418Z inflating: build/bin/broadcast_test 2023-11-14T04:54:37.9625835Z inflating: build/bin/basic 2023-11-14T04:54:37.9692357Z inflating: build/bin/cpu_profiling_allocator_test 2023-11-14T04:54:37.9765246Z inflating: build/bin/cpu_generator_test 2023-11-14T04:54:37.9827810Z inflating: build/bin/dispatch_key_set_test 2023-11-14T04:54:37.9944124Z inflating: build/bin/cpu_rng_test 2023-11-14T04:54:38.0007280Z inflating: build/bin/dlconvertor_test 2023-11-14T04:54:38.0079820Z inflating: build/bin/extension_backend_test 2023-11-14T04:54:38.0149438Z inflating: build/bin/half_test 2023-11-14T04:54:38.0267848Z inflating: build/bin/ivalue_test 2023-11-14T04:54:38.0336205Z inflating: build/bin/math_kernel_test 2023-11-14T04:54:38.0398701Z inflating: build/bin/lazy_tensor_test 2023-11-14T04:54:38.0466487Z inflating: build/bin/memory_format_test 2023-11-14T04:54:38.0532865Z inflating: build/bin/memory_overlapping_test 2023-11-14T04:54:38.0596038Z inflating: build/bin/operator_name_test 2023-11-14T04:54:38.0662805Z inflating: build/bin/mobile_memory_cleanup 2023-11-14T04:54:38.0732987Z inflating: build/bin/native_test 2023-11-14T04:54:38.0797644Z inflating: build/bin/operators_test 2023-11-14T04:54:38.0862798Z inflating: build/bin/packedtensoraccessor_test 2023-11-14T04:54:38.0945653Z inflating: build/bin/pow_test 2023-11-14T04:54:38.1017232Z inflating: build/bin/quantized_test 2023-11-14T04:54:38.1081035Z inflating: build/bin/reportMemoryUsage_test 2023-11-14T04:54:38.1143505Z inflating: build/bin/reduce_ops_test 2023-11-14T04:54:38.1216088Z inflating: build/bin/scalar_test 2023-11-14T04:54:38.1287428Z inflating: build/bin/scalar_tensor_test 2023-11-14T04:54:38.1352081Z inflating: build/bin/StorageUtils_test 2023-11-14T04:54:38.1422287Z inflating: build/bin/type_ptr_test 2023-11-14T04:54:38.1488396Z inflating: build/bin/stride_properties_test 2023-11-14T04:54:38.1585838Z inflating: build/bin/tensor_iterator_test 2023-11-14T04:54:38.1589216Z inflating: build/bin/thread_init_test 2023-11-14T04:54:38.1657625Z inflating: build/bin/test_parallel 2023-11-14T04:54:38.1733623Z inflating: build/bin/type_test 2023-11-14T04:54:38.1735646Z inflating: build/bin/verify_api_visibility 2023-11-14T04:54:38.1801404Z inflating: build/bin/undefined_tensor_test 2023-11-14T04:54:38.1888319Z inflating: build/bin/legacy_vmap_test 2023-11-14T04:54:38.1953196Z inflating: build/bin/weakref_test 2023-11-14T04:54:38.2017954Z inflating: build/bin/wrapdim_test 2023-11-14T04:54:38.2151899Z inflating: build/bin/List_test 2023-11-14T04:54:38.2214003Z inflating: build/bin/xla_tensor_test 2023-11-14T04:54:38.2289004Z inflating: build/bin/IListRef_test 2023-11-14T04:54:38.2441947Z inflating: build/bin/kernel_function_legacy_test 2023-11-14T04:54:38.2524977Z inflating: build/bin/KernelFunction_test 2023-11-14T04:54:38.2646413Z inflating: build/bin/kernel_function_test 2023-11-14T04:54:38.2806554Z inflating: build/bin/kernel_lambda_legacy_test 2023-11-14T04:54:38.2883035Z inflating: build/bin/kernel_stackbased_test 2023-11-14T04:54:38.3012782Z inflating: build/bin/kernel_lambda_test 2023-11-14T04:54:38.3076946Z inflating: build/bin/CppSignature_test 2023-11-14T04:54:38.3198549Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2023-11-14T04:54:38.3259892Z inflating: build/bin/op_allowlist_test 2023-11-14T04:54:38.3329328Z inflating: build/bin/backend_fallback_test 2023-11-14T04:54:38.3409293Z inflating: build/bin/inline_container_test 2023-11-14T04:54:38.3788140Z inflating: build/bin/op_registration_test 2023-11-14T04:54:38.3854814Z inflating: build/bin/cuda_apply_test 2023-11-14T04:54:38.3928571Z inflating: build/bin/cuda_atomic_ops_test 2023-11-14T04:54:38.3997250Z inflating: build/bin/cuda_caching_host_allocator_test 2023-11-14T04:54:38.4059761Z inflating: build/bin/cuda_device_test 2023-11-14T04:54:38.4147603Z inflating: build/bin/cuda_complex_math_test 2023-11-14T04:54:38.4222009Z inflating: build/bin/cuda_complex_test 2023-11-14T04:54:38.4294695Z inflating: build/bin/cuda_cub_test 2023-11-14T04:54:38.4358365Z inflating: build/bin/cuda_dlconvertor_test 2023-11-14T04:54:38.4422720Z inflating: build/bin/cuda_integer_divider_test 2023-11-14T04:54:38.4504049Z inflating: build/bin/cuda_distributions_test 2023-11-14T04:54:38.4566596Z inflating: build/bin/cuda_half_test 2023-11-14T04:54:38.4637883Z inflating: build/bin/cuda_generator_test 2023-11-14T04:54:38.4700126Z inflating: build/bin/cuda_optional_test 2023-11-14T04:54:38.4762752Z inflating: build/bin/cuda_allocatorTraceTracker_test 2023-11-14T04:54:38.4828868Z inflating: build/bin/cuda_reportMemoryUsage_test 2023-11-14T04:54:38.4904784Z inflating: build/bin/cuda_stream_test 2023-11-14T04:54:38.4969962Z inflating: build/bin/cuda_packedtensoraccessor_test 2023-11-14T04:54:38.5031816Z inflating: build/bin/cuda_cudnn_test 2023-11-14T04:54:38.5097218Z inflating: build/bin/cuda_vectorized_test 2023-11-14T04:54:38.5116091Z inflating: build/bin/tutorial_tensorexpr 2023-11-14T04:54:38.5197982Z inflating: build/bin/ProcessGroupGlooTest 2023-11-14T04:54:38.5269890Z inflating: build/bin/ProcessGroupGlooAsyncTest 2023-11-14T04:54:38.5348471Z inflating: build/bin/ProcessGroupNCCLTest 2023-11-14T04:54:38.5422998Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2023-11-14T04:54:38.5488803Z inflating: build/bin/ProcessGroupUCCTest 2023-11-14T04:54:38.5557743Z inflating: build/bin/test_dist_autograd 2023-11-14T04:54:38.5644563Z inflating: build/bin/test_cpp_rpc 2023-11-14T04:54:38.5647868Z inflating: build/bin/parallel_benchmark 2023-11-14T04:54:38.5659389Z inflating: build/bin/aot_model_compiler_test 2023-11-14T04:54:38.5743354Z inflating: build/bin/test_mobile_nnc 2023-11-14T04:54:38.6177495Z inflating: build/bin/test_lazy 2023-11-14T04:54:38.7218252Z inflating: build/bin/test_tensorexpr 2023-11-14T04:54:38.7223802Z inflating: build/bin/torch_shm_manager 2023-11-14T04:54:38.8681687Z inflating: build/bin/test_api 2023-11-14T04:54:38.9384782Z inflating: build/bin/test_jit 2023-11-14T04:54:38.9385363Z creating: .additional_ci_files/ 2023-11-14T04:54:38.9419930Z inflating: .additional_ci_files/test-times.json 2023-11-14T04:54:38.9558420Z inflating: .additional_ci_files/test-class-times.json 2023-11-14T04:54:39.0445175Z inflating: .additional_ci_files/test-file-ratings.json 2023-11-14T04:54:39.2042483Z inflating: .additional_ci_files/test-class-ratings.json 2023-11-14T04:54:39.2244388Z inflating: .additional_ci_files/td_heuristic_historical_edited_files.json 2023-11-14T04:54:39.2336702Z inflating: .additional_ci_files/td_heuristic_profiling.json 2023-11-14T04:54:39.2375450Z ##[group]Run df -H 2023-11-14T04:54:39.2375822Z df -H 2023-11-14T04:54:39.2384417Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:39.2385044Z env: 2023-11-14T04:54:39.2385381Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:39.2385926Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:39.2386529Z ##[endgroup] 2023-11-14T04:54:39.2424898Z Filesystem Size Used Avail Use% Mounted on 2023-11-14T04:54:39.2425636Z devtmpfs 129G 0 129G 0% /dev 2023-11-14T04:54:39.2426875Z tmpfs 129G 8.7M 129G 1% /dev/shm 2023-11-14T04:54:39.2428101Z tmpfs 129G 553k 129G 1% /run 2023-11-14T04:54:39.2428709Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2023-11-14T04:54:39.2429301Z /dev/xvda1 162G 39G 123G 25% / 2023-11-14T04:54:39.2466836Z ##[group]Run .github/scripts/parse_ref.py 2023-11-14T04:54:39.2467452Z .github/scripts/parse_ref.py 2023-11-14T04:54:39.2475283Z shell: /usr/bin/bash -e {0} 2023-11-14T04:54:39.2475703Z env: 2023-11-14T04:54:39.2476037Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:39.2476604Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:39.2477189Z ##[endgroup] 2023-11-14T04:54:39.2787720Z Prepare all required actions 2023-11-14T04:54:39.2838745Z ##[group]Run ./.github/actions/get-workflow-job-id 2023-11-14T04:54:39.2839315Z with: 2023-11-14T04:54:39.2840008Z github-token: *** 2023-11-14T04:54:39.2840390Z env: 2023-11-14T04:54:39.2840703Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:39.2841265Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:39.2841871Z ##[endgroup] 2023-11-14T04:54:39.2865687Z ##[group]Run set -eux 2023-11-14T04:54:39.2866110Z set -eux 2023-11-14T04:54:39.2866847Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2023-11-14T04:54:39.2875032Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:39.2875661Z env: 2023-11-14T04:54:39.2876002Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:39.2876558Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:39.2877381Z GITHUB_TOKEN: *** 2023-11-14T04:54:39.2877770Z ##[endgroup] 2023-11-14T04:54:39.2905101Z + python3 .github/scripts/get_workflow_job_id.py 6859165800 i-0495cce9e02a09ed6 2023-11-14T04:54:42.7205129Z setting job-id=18651430019 2023-11-14T04:54:42.7206327Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:54:42.7449297Z Prepare all required actions 2023-11-14T04:54:42.7449906Z Getting action download info 2023-11-14T04:54:42.8688018Z ##[group]Run ./.github/actions/filter-test-configs 2023-11-14T04:54:42.8688582Z with: 2023-11-14T04:54:42.8689174Z github-token: *** 2023-11-14T04:54:42.8691309Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2023-11-14T04:54:42.8693697Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:54:42.8694620Z env: 2023-11-14T04:54:42.8694944Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:42.8695518Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:42.8696153Z ##[endgroup] 2023-11-14T04:54:43.0097118Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T04:54:43.0097811Z with: 2023-11-14T04:54:43.0098139Z shell: bash 2023-11-14T04:54:43.0098509Z timeout_minutes: 10 2023-11-14T04:54:43.0098892Z max_attempts: 5 2023-11-14T04:54:43.0099278Z retry_wait_seconds: 30 2023-11-14T04:54:43.0099927Z command: set -eux python3 -m pip install requests==2.26.0 pyyaml==6.0 2023-11-14T04:54:43.0100650Z polling_interval_seconds: 1 2023-11-14T04:54:43.0101116Z warning_on_retry: true 2023-11-14T04:54:43.0101552Z continue_on_error: false 2023-11-14T04:54:43.0101948Z env: 2023-11-14T04:54:43.0102283Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:43.0102840Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:43.0103683Z GITHUB_TOKEN: *** 2023-11-14T04:54:43.0104057Z ##[endgroup] 2023-11-14T04:54:43.0832953Z + python3 -m pip install requests==2.26.0 pyyaml==6.0 2023-11-14T04:54:43.3703124Z Defaulting to user installation because normal site-packages is not writeable 2023-11-14T04:54:43.3913515Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0) 2023-11-14T04:54:43.4093100Z Requirement already satisfied: pyyaml==6.0 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0) 2023-11-14T04:54:43.4105404Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4) 2023-11-14T04:54:43.4121479Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2023.7.22) 2023-11-14T04:54:43.4133021Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.18) 2023-11-14T04:54:43.4373195Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12) 2023-11-14T04:54:44.0834248Z Command completed after 1 attempt(s). 2023-11-14T04:54:44.1022442Z ##[group]Run set -x 2023-11-14T04:54:44.1023256Z set -x 2023-11-14T04:54:44.1023996Z  2023-11-14T04:54:44.1025300Z # Use relative path here as this could be checked out anywhere, not necessarily 2023-11-14T04:54:44.1027017Z # in runner workspace 2023-11-14T04:54:44.1028326Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2023-11-14T04:54:44.1042773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:44.1044005Z env: 2023-11-14T04:54:44.1044623Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:44.1045741Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:44.1046948Z ##[endgroup] 2023-11-14T04:54:44.1091369Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2023-11-14T04:54:44.2308696Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2023-11-14T04:54:44.2309569Z echo "Workflow: ${GITHUB_WORKFLOW}" 2023-11-14T04:54:44.2310176Z echo "Job name: ${JOB_NAME}" 2023-11-14T04:54:44.2310670Z  2023-11-14T04:54:44.2311334Z # Use relative path here as this could be checked out anywhere, not necessarily 2023-11-14T04:54:44.2312184Z # in runner workspace 2023-11-14T04:54:44.2312910Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2023-11-14T04:54:44.2313724Z  --workflow "${GITHUB_WORKFLOW}" \ 2023-11-14T04:54:44.2314274Z  --job-name "${JOB_NAME}" \ 2023-11-14T04:54:44.2316285Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2023-11-14T04:54:44.2318332Z  --pr-number "${PR_NUMBER}" \ 2023-11-14T04:54:44.2318956Z  --tag "${TAG}" \ 2023-11-14T04:54:44.2319422Z  --event-name "${EVENT_NAME}" \ 2023-11-14T04:54:44.2319981Z  --schedule "${SCHEDULE}" \ 2023-11-14T04:54:44.2320506Z  --branch "${HEAD_BRANCH}" 2023-11-14T04:54:44.2329012Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:44.2329617Z env: 2023-11-14T04:54:44.2329958Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:44.2330525Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:44.2331917Z GITHUB_TOKEN: *** 2023-11-14T04:54:44.2332741Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:54:44.2333665Z PR_NUMBER: 112098 2023-11-14T04:54:44.2334029Z TAG: 2023-11-14T04:54:44.2334383Z EVENT_NAME: pull_request 2023-11-14T04:54:44.2334807Z SCHEDULE: 2023-11-14T04:54:44.2335139Z HEAD_BRANCH: 2023-11-14T04:54:44.2335740Z ##[endgroup] 2023-11-14T04:54:44.2363884Z Workflow: pull 2023-11-14T04:54:44.2365676Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:54:44.6046597Z fatal: unknown commit main 2023-11-14T04:54:44.6073862Z /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/filter_test_configs.py:472: UserWarning: failed to get commit messages: Command '['git', 'cherry', '-v', 'main']' returned non-zero exit status 128. 2023-11-14T04:54:44.6075884Z warnings.warn(f"failed to get commit messages: {e}") 2023-11-14T04:54:44.6235386Z ##[group]Run echo "Filtered matrix:" 2023-11-14T04:54:44.6236455Z echo "Filtered matrix:" 2023-11-14T04:54:44.6240370Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2023-11-14T04:54:44.6244319Z  2023-11-14T04:54:44.6244990Z echo 2023-11-14T04:54:44.6245904Z echo "Is the current job unstable? False" 2023-11-14T04:54:44.6246998Z  2023-11-14T04:54:44.6247607Z echo 2023-11-14T04:54:44.6248408Z echo "Is keep-going label set? True" 2023-11-14T04:54:44.6249423Z  2023-11-14T04:54:44.6250033Z echo 2023-11-14T04:54:44.6251087Z echo "Renabled issues? " 2023-11-14T04:54:44.6263709Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:44.6265156Z env: 2023-11-14T04:54:44.6265791Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:44.6266885Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:44.6267525Z ##[endgroup] 2023-11-14T04:54:44.6300789Z Filtered matrix: 2023-11-14T04:54:44.6304880Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2023-11-14T04:54:44.6308313Z 2023-11-14T04:54:44.6308699Z Is the current job unstable? False 2023-11-14T04:54:44.6309433Z 2023-11-14T04:54:44.6309934Z Is keep-going label set? True 2023-11-14T04:54:44.6310575Z 2023-11-14T04:54:44.6310902Z Renabled issues? 2023-11-14T04:54:44.6425732Z Prepare all required actions 2023-11-14T04:54:44.6426756Z Getting action download info 2023-11-14T04:54:44.7641201Z ##[group]Run ./.github/actions/pytest-cache-download 2023-11-14T04:54:44.7641767Z with: 2023-11-14T04:54:44.7642115Z cache_dir: .pytest_cache 2023-11-14T04:54:44.7642788Z job_identifier: pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed 2023-11-14T04:54:44.7643475Z env: 2023-11-14T04:54:44.7643809Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:44.7644367Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:44.7644963Z ##[endgroup] 2023-11-14T04:54:44.8897936Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T04:54:44.8898648Z with: 2023-11-14T04:54:44.8898964Z shell: bash 2023-11-14T04:54:44.8899337Z timeout_minutes: 5 2023-11-14T04:54:44.8899740Z max_attempts: 5 2023-11-14T04:54:44.8900113Z retry_wait_seconds: 30 2023-11-14T04:54:44.8900746Z command: set -eu python3 -m pip install boto3==1.19.12 2023-11-14T04:54:44.8901373Z polling_interval_seconds: 1 2023-11-14T04:54:44.8901839Z warning_on_retry: true 2023-11-14T04:54:44.8902276Z continue_on_error: false 2023-11-14T04:54:44.8902674Z env: 2023-11-14T04:54:44.8903012Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:44.8903574Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:44.8904159Z ##[endgroup] 2023-11-14T04:54:45.2511063Z Defaulting to user installation because normal site-packages is not writeable 2023-11-14T04:54:45.2712517Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12) 2023-11-14T04:54:45.2782885Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2) 2023-11-14T04:54:45.2819613Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12) 2023-11-14T04:54:45.2878483Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0) 2023-11-14T04:54:45.2895696Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2) 2023-11-14T04:54:45.2929355Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18) 2023-11-14T04:54:45.3167771Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2023-11-14T04:54:45.9625097Z Command completed after 1 attempt(s). 2023-11-14T04:54:45.9679671Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2023-11-14T04:54:45.9680418Z python3 .github/scripts/pytest_cache.py \ 2023-11-14T04:54:45.9681006Z  --download \ 2023-11-14T04:54:45.9681555Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2023-11-14T04:54:45.9682172Z  --pr_identifier $GITHUB_REF \ 2023-11-14T04:54:45.9682762Z  --job_identifier $JOB_IDENTIFIER \ 2023-11-14T04:54:45.9683338Z  --temp_dir $RUNNER_TEMP \ 2023-11-14T04:54:45.9683826Z  --repo $REPO \ 2023-11-14T04:54:45.9692835Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:45.9693458Z env: 2023-11-14T04:54:45.9693793Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:45.9694336Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:45.9694976Z CACHE_DIR: .pytest_cache 2023-11-14T04:54:45.9695645Z JOB_IDENTIFIER: pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed 2023-11-14T04:54:45.9696360Z REPO: pytorch/pytorch 2023-11-14T04:54:45.9696770Z ##[endgroup] 2023-11-14T04:54:46.4260142Z PR identifier for `refs/pull/112098/merge` is `2e5132e7254b4b60102d80422aad461b` 2023-11-14T04:54:46.4264165Z Downloading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=True, job_identifier='pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed', pr_identifier='refs/pull/112098/merge', repo='pytorch/pytorch', shard=None, temp_dir='/home/ec2-user/actions-runner/_work/_temp', upload=False) 2023-11-14T04:54:46.4267843Z Downloading s3://gha-artifacts/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1.zip 2023-11-14T04:54:46.4271302Z to /home/ec2-user/actions-runner/_work/_temp/cache-zip-downloads/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1.zip 2023-11-14T04:54:46.4274617Z Downloading s3://gha-artifacts/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2.zip 2023-11-14T04:54:46.4278062Z to /home/ec2-user/actions-runner/_work/_temp/cache-zip-downloads/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2.zip 2023-11-14T04:54:46.4282588Z Unzipping /home/ec2-user/actions-runner/_work/_temp/cache-zip-downloads/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1.zip 2023-11-14T04:54:46.4286487Z to /home/ec2-user/actions-runner/_work/_temp/unzipped-caches/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1/.pytest_cache 2023-11-14T04:54:46.4289138Z Merging cache for job_identifier `pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed`, shard `1` into `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` 2023-11-14T04:54:46.4293033Z Unzipping /home/ec2-user/actions-runner/_work/_temp/cache-zip-downloads/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2.zip 2023-11-14T04:54:46.4296720Z to /home/ec2-user/actions-runner/_work/_temp/unzipped-caches/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2/.pytest_cache 2023-11-14T04:54:46.4299370Z Merging cache for job_identifier `pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed`, shard `2` into `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` 2023-11-14T04:54:46.4665234Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2023-11-14T04:54:46.4666151Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2023-11-14T04:54:46.4675028Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:46.4675648Z env: 2023-11-14T04:54:46.4675967Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:46.4676530Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:46.4677131Z JOB_TIMEOUT: 360 2023-11-14T04:54:46.4677485Z ##[endgroup] 2023-11-14T04:54:46.4777999Z ##[group]Run set -x 2023-11-14T04:54:46.4778539Z set -x 2023-11-14T04:54:46.4778901Z  2023-11-14T04:54:46.4779337Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2023-11-14T04:54:46.4780013Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2023-11-14T04:54:46.4780717Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2023-11-14T04:54:46.4781349Z  TEST_COMMAND=.ci/onnx/test.sh 2023-11-14T04:54:46.4781841Z else 2023-11-14T04:54:46.4782259Z  TEST_COMMAND=.ci/pytorch/test.sh 2023-11-14T04:54:46.4782783Z fi 2023-11-14T04:54:46.4783099Z  2023-11-14T04:54:46.4783676Z # detached container should get cleaned up by teardown_ec2_linux 2023-11-14T04:54:46.4784627Z # TODO: Stop building test binaries as part of the build phase 2023-11-14T04:54:46.4785459Z # Used for GPU_FLAG since that doesn't play nice 2023-11-14T04:54:46.4786194Z # shellcheck disable=SC2086,SC2090 2023-11-14T04:54:46.4786751Z container_name=$(docker run \ 2023-11-14T04:54:46.4787266Z  ${GPU_FLAG:-} \ 2023-11-14T04:54:46.4787718Z  -e BUILD_ENVIRONMENT \ 2023-11-14T04:54:46.4788201Z  -e PR_NUMBER \ 2023-11-14T04:54:46.4788648Z  -e GITHUB_ACTIONS \ 2023-11-14T04:54:46.4789129Z  -e GITHUB_REPOSITORY \ 2023-11-14T04:54:46.4789611Z  -e GITHUB_WORKFLOW \ 2023-11-14T04:54:46.4790086Z  -e GITHUB_JOB \ 2023-11-14T04:54:46.4790528Z  -e GITHUB_RUN_ID \ 2023-11-14T04:54:46.4790995Z  -e GITHUB_RUN_NUMBER \ 2023-11-14T04:54:46.4791473Z  -e GITHUB_RUN_ATTEMPT \ 2023-11-14T04:54:46.4791960Z  -e JOB_ID \ 2023-11-14T04:54:46.4792375Z  -e BASE_SHA \ 2023-11-14T04:54:46.4792776Z  -e BRANCH \ 2023-11-14T04:54:46.4793181Z  -e SHA1 \ 2023-11-14T04:54:46.4793596Z  -e AWS_DEFAULT_REGION \ 2023-11-14T04:54:46.4794077Z  -e IN_WHEEL_TEST \ 2023-11-14T04:54:46.4794540Z  -e SHARD_NUMBER \ 2023-11-14T04:54:46.4794990Z  -e TEST_CONFIG \ 2023-11-14T04:54:46.4795422Z  -e NUM_TEST_SHARDS \ 2023-11-14T04:54:46.4796104Z  -e REENABLED_ISSUES \ 2023-11-14T04:54:46.4796607Z  -e CONTINUE_THROUGH_ERROR \ 2023-11-14T04:54:46.4797136Z  -e PYTORCH_RETRY_TEST_CASES \ 2023-11-14T04:54:46.4797722Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2023-11-14T04:54:46.4798276Z  -e PR_LABELS \ 2023-11-14T04:54:46.4798758Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2023-11-14T04:54:46.4799317Z  -e SCCACHE_BUCKET \ 2023-11-14T04:54:46.4799801Z  -e SCCACHE_S3_KEY_PREFIX \ 2023-11-14T04:54:46.4800286Z  -e XLA_CUDA \ 2023-11-14T04:54:46.4800776Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2023-11-14T04:54:46.4801400Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2023-11-14T04:54:46.4802034Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2023-11-14T04:54:46.4802669Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2023-11-14T04:54:46.4803248Z  -e HUGGING_FACE_HUB_TOKEN \ 2023-11-14T04:54:46.4803754Z  -e DASHBOARD_TAG \ 2023-11-14T04:54:46.4804341Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2023-11-14T04:54:46.4805001Z  --ulimit stack=10485760:83886080 \ 2023-11-14T04:54:46.4805605Z  --security-opt seccomp=unconfined \ 2023-11-14T04:54:46.4806162Z  --cap-add=SYS_PTRACE \ 2023-11-14T04:54:46.4806646Z  --ipc=host \ 2023-11-14T04:54:46.4807080Z  --shm-size="${SHM_SIZE}" \ 2023-11-14T04:54:46.4807550Z  --tty \ 2023-11-14T04:54:46.4807932Z  --detach \ 2023-11-14T04:54:46.4808366Z  --name="${container_name}" \ 2023-11-14T04:54:46.4808861Z  --user jenkins \ 2023-11-14T04:54:46.4809465Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2023-11-14T04:54:46.4810273Z  -w /var/lib/jenkins/workspace \ 2023-11-14T04:54:46.4811211Z  "${DOCKER_IMAGE}" 2023-11-14T04:54:46.4811658Z ) 2023-11-14T04:54:46.4812157Z # Propagate download.pytorch.org IP to container 2023-11-14T04:54:46.4813309Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2023-11-14T04:54:46.4814529Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2023-11-14T04:54:46.4815680Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2023-11-14T04:54:46.4824137Z shell: /usr/bin/bash -e {0} 2023-11-14T04:54:46.4824571Z env: 2023-11-14T04:54:46.4824903Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:46.4825443Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:46.4826167Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:54:46.4826765Z PR_NUMBER: 112098 2023-11-14T04:54:46.4827172Z GITHUB_REPOSITORY: pytorch/pytorch 2023-11-14T04:54:46.4827678Z GITHUB_WORKFLOW: pull 2023-11-14T04:54:46.4828087Z GITHUB_JOB: test 2023-11-14T04:54:46.4828449Z GITHUB_RUN_ID: 6859165800 2023-11-14T04:54:46.4828897Z GITHUB_RUN_NUMBER: 155684 2023-11-14T04:54:46.4829335Z GITHUB_RUN_ATTEMPT: 1 2023-11-14T04:54:46.4829720Z JOB_ID: 18651430019 2023-11-14T04:54:46.4830116Z BRANCH: pull/112098 2023-11-14T04:54:46.4830577Z SHA1: fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:54:46.4831198Z BASE_SHA: e1c872e009e8aa07281f38723353dc43baf2fc12 2023-11-14T04:54:46.4831788Z PYTORCH_RETRY_TEST_CASES: 1 2023-11-14T04:54:46.4832265Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2023-11-14T04:54:46.4832738Z TEST_CONFIG: distributed 2023-11-14T04:54:46.4833157Z SHARD_NUMBER: 1 2023-11-14T04:54:46.4833528Z NUM_TEST_SHARDS: 3 2023-11-14T04:54:46.4833904Z REENABLED_ISSUES: 2023-11-14T04:54:46.4834312Z CONTINUE_THROUGH_ERROR: True 2023-11-14T04:54:46.4834870Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2023-11-14T04:54:46.4835458Z SCCACHE_S3_KEY_PREFIX: pull 2023-11-14T04:54:46.4835896Z SHM_SIZE: 2g 2023-11-14T04:54:46.4837107Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:54:46.4838613Z XLA_CUDA: 2023-11-14T04:54:46.4839187Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2023-11-14T04:54:46.4839950Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2023-11-14T04:54:46.4840483Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2023-11-14T04:54:46.4840964Z DASHBOARD_TAG: 2023-11-14T04:54:46.4841346Z HUGGING_FACE_HUB_TOKEN: 2023-11-14T04:54:46.4841763Z ##[endgroup] 2023-11-14T04:54:46.4868834Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2023-11-14T04:54:46.4869587Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2023-11-14T04:54:46.4870199Z + TEST_COMMAND=.ci/pytorch/test.sh 2023-11-14T04:54:46.4879425Z +++ nproc --ignore=2 2023-11-14T04:54:46.4898358Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_6859165800 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:54:57.2288100Z + container_name=99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T04:54:57.2290495Z + grep download.pytorch.org /etc/hosts 2023-11-14T04:54:57.2292825Z + docker exec -i 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe sudo bash -c '/bin/cat >> /etc/hosts' 2023-11-14T04:54:57.3080014Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2023-11-14T04:54:57.3129599Z + echo DOCKER_CONTAINER_ID=99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T04:54:57.3135300Z ++ echo dist/torch-2.2.0a0+gitfe705c0-cp310-cp310-linux_x86_64.whl 2023-11-14T04:54:57.3138203Z + docker exec -t 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe sh -c 'pip install dist/torch-2.2.0a0+gitfe705c0-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2023-11-14T04:54:57.8549694Z Processing ./dist/torch-2.2.0a0+gitfe705c0-cp310-cp310-linux_x86_64.whl 2023-11-14T04:54:58.9386797Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (3.9.0) 2023-11-14T04:54:58.9389718Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (4.8.0) 2023-11-14T04:54:58.9394564Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (1.12) 2023-11-14T04:54:58.9399287Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (2.8.8) 2023-11-14T04:54:58.9403904Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (3.1.2) 2023-11-14T04:54:58.9408714Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (2023.4.0) 2023-11-14T04:54:58.9425300Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (3.3.0) 2023-11-14T04:54:58.9507862Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.2.0a0+gitfe705c0) (1.21.2) 2023-11-14T04:54:59.0357285Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.2.0a0+gitfe705c0) (2.1.3) 2023-11-14T04:54:59.0581369Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.2.0a0+gitfe705c0) (1.3.0) 2023-11-14T04:55:00.1173619Z Installing collected packages: torch 2023-11-14T04:55:11.3879682Z Successfully installed torch-2.2.0a0+gitfe705c0 2023-11-14T04:55:11.4626303Z + echo 'Environment variables:' 2023-11-14T04:55:11.4628360Z Environment variables: 2023-11-14T04:55:11.4628850Z + env 2023-11-14T04:55:11.4634649Z INSTALLED_DB=yes 2023-11-14T04:55:11.4635194Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2023-11-14T04:55:11.4635692Z NVIDIA_VISIBLE_DEVICES=all 2023-11-14T04:55:11.4638723Z NV_NVML_DEV_VERSION=11.8.86-1 2023-11-14T04:55:11.4640098Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:55:11.4640838Z CONTINUE_THROUGH_ERROR=True 2023-11-14T04:55:11.4641456Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2023-11-14T04:55:11.4642120Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2023-11-14T04:55:11.4642784Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:55:11.4643405Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2023-11-14T04:55:11.4643879Z HOSTNAME=99f90b6e6ff3 2023-11-14T04:55:11.4644924Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.4645978Z GITHUB_ACTION=__self 2023-11-14T04:55:11.4646407Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2023-11-14T04:55:11.4654578Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 2023-11-14T04:55:11.4662562Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2023-11-14T04:55:11.4663205Z NV_NVTX_VERSION=11.8.86-1 2023-11-14T04:55:11.4663641Z GITHUB_RUN_NUMBER=155684 2023-11-14T04:55:11.4664050Z TEST_CONFIG=distributed 2023-11-14T04:55:11.4664489Z GITHUB_REPOSITORY_OWNER_ID=21003710 2023-11-14T04:55:11.4665076Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2023-11-14T04:55:11.4665626Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2023-11-14T04:55:11.4666173Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2023-11-14T04:55:11.4666693Z NV_LIBNPP_VERSION=11.8.0.86-1 2023-11-14T04:55:11.4667143Z GITHUB_TRIGGERING_ACTOR=clee2000 2023-11-14T04:55:11.4667769Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2023-11-14T04:55:11.4668352Z GITHUB_REF_TYPE=branch 2023-11-14T04:55:11.4668773Z TORCH_CUDA_ARCH_LIST=Maxwell 2023-11-14T04:55:11.4669235Z NCCL_VERSION=2.16.2-1 2023-11-14T04:55:11.4669713Z BASE_SHA=e1c872e009e8aa07281f38723353dc43baf2fc12 2023-11-14T04:55:11.4670259Z XLA_CUDA= 2023-11-14T04:55:11.4670583Z HUGGING_FACE_HUB_TOKEN= 2023-11-14T04:55:11.4673035Z *** 2023-11-14T04:55:11.4673403Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2023-11-14T04:55:11.4673886Z GITHUB_REPOSITORY_ID=65600975 2023-11-14T04:55:11.4674337Z GITHUB_ACTIONS=true 2023-11-14T04:55:11.4674744Z NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:55:11.4675362Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2023-11-14T04:55:11.4676024Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2023-11-14T04:55:11.4676603Z SHA1=fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:55:11.4677213Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2023-11-14T04:55:11.4677823Z GITHUB_SHA=35b9c00c6974fbcfd0eb53491c53c61a6c439831 2023-11-14T04:55:11.4678716Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/112098/merge 2023-11-14T04:55:11.4679501Z UCC_HOME=/usr 2023-11-14T04:55:11.4679928Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2023-11-14T04:55:11.4680433Z NVIDIA_PRODUCT_NAME=CUDA 2023-11-14T04:55:11.4680985Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2023-11-14T04:55:11.4681579Z GITHUB_REF=refs/pull/112098/merge 2023-11-14T04:55:11.4682094Z NV_CUDA_CUDART_VERSION=11.8.89-1 2023-11-14T04:55:11.4682537Z SHARD_NUMBER=1 2023-11-14T04:55:11.4682914Z GITHUB_REF_PROTECTED=false 2023-11-14T04:55:11.4683343Z HOME=/var/lib/jenkins 2023-11-14T04:55:11.4683776Z GITHUB_API_URL=https://api.github.com 2023-11-14T04:55:11.4684326Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2023-11-14T04:55:11.4684916Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2023-11-14T04:55:11.4685489Z SCCACHE_S3_KEY_PREFIX=pull 2023-11-14T04:55:11.4685918Z CUDA_VERSION=11.8.0 2023-11-14T04:55:11.4686444Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2023-11-14T04:55:11.4686974Z NUM_TEST_SHARDS=3 2023-11-14T04:55:11.4687344Z UCX_HOME=/usr 2023-11-14T04:55:11.4707806Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2023-11-14T04:55:11.4709265Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.4710339Z PYTORCH_RETRY_TEST_CASES=1 2023-11-14T04:55:11.4711399Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.4712825Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2023-11-14T04:55:11.4713674Z GITHUB_EVENT_NAME=pull_request 2023-11-14T04:55:11.4714117Z DASHBOARD_TAG= 2023-11-14T04:55:11.4714495Z GITHUB_RUN_ID=6859165800 2023-11-14T04:55:11.4715071Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2023-11-14T04:55:11.4715726Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2023-11-14T04:55:11.4716988Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.4718094Z GITHUB_ACTOR=clee2000 2023-11-14T04:55:11.4718557Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2023-11-14T04:55:11.4719020Z PR_NUMBER=112098 2023-11-14T04:55:11.4719404Z GITHUB_RUN_ATTEMPT=1 2023-11-14T04:55:11.4719826Z ANACONDA_PYTHON_VERSION=3.10 2023-11-14T04:55:11.4720380Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2023-11-14T04:55:11.4720960Z TERM=xterm 2023-11-14T04:55:11.4721375Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2023-11-14T04:55:11.4721952Z INSTALLED_VISION=yes 2023-11-14T04:55:11.4722355Z BRANCH=pull/112098 2023-11-14T04:55:11.4722742Z OPENSSL_ROOT_DIR=/opt/openssl 2023-11-14T04:55:11.4723240Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2023-11-14T04:55:11.4723773Z CUDA_PATH=/usr/local/cuda 2023-11-14T04:55:11.4724712Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2023-11-14T04:55:11.4725670Z GITHUB_SERVER_URL=https://github.com 2023-11-14T04:55:11.4726282Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2023-11-14T04:55:11.4726875Z REENABLED_ISSUES= 2023-11-14T04:55:11.4727236Z SHLVL=1 2023-11-14T04:55:11.4727564Z MAX_JOBS=30 2023-11-14T04:55:11.4727942Z NV_CUDA_LIB_VERSION=11.8.0-1 2023-11-14T04:55:11.4728384Z NVARCH=x86_64 2023-11-14T04:55:11.4728753Z GITHUB_ACTOR_ID=44682903 2023-11-14T04:55:11.4729479Z GITHUB_WORKFLOW_SHA=35b9c00c6974fbcfd0eb53491c53c61a6c439831 2023-11-14T04:55:11.4730130Z GITHUB_REF_NAME=112098/merge 2023-11-14T04:55:11.4731022Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2023-11-14T04:55:11.4731870Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2023-11-14T04:55:11.4732601Z GITHUB_JOB=test 2023-11-14T04:55:11.4733103Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2023-11-14T04:55:11.4733803Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-11-14T04:55:11.4734495Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2023-11-14T04:55:11.4735048Z GITHUB_REPOSITORY=pytorch/pytorch 2023-11-14T04:55:11.4735570Z NV_NVPROF_VERSION=11.8.87-1 2023-11-14T04:55:11.4736002Z GITHUB_RETENTION_DAYS=90 2023-11-14T04:55:11.4736440Z OPENSSL_DIR=/opt/openssl 2023-11-14T04:55:11.4736893Z GITHUB_ACTION_REPOSITORY= 2023-11-14T04:55:11.4738213Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-11-14T04:55:11.4739598Z GITHUB_BASE_REF=main 2023-11-14T04:55:11.4740026Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2023-11-14T04:55:11.4740469Z CI=true 2023-11-14T04:55:11.4740859Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2023-11-14T04:55:11.4741384Z GITHUB_REPOSITORY_OWNER=pytorch 2023-11-14T04:55:11.4741832Z JOB_ID=18651430019 2023-11-14T04:55:11.4742221Z INSTALLED_PROTOBUF=yes 2023-11-14T04:55:11.4742640Z GITHUB_HEAD_REF=csl/test3 2023-11-14T04:55:11.4743044Z GITHUB_ACTION_REF= 2023-11-14T04:55:11.4743583Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2023-11-14T04:55:11.4744171Z GITHUB_WORKFLOW=pull 2023-11-14T04:55:11.4744580Z DEBIAN_FRONTEND=noninteractive 2023-11-14T04:55:11.4745860Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.4746960Z SKIP_SCCACHE_INITIALIZATION=1 2023-11-14T04:55:11.4747399Z _=/usr/bin/env 2023-11-14T04:55:11.4748001Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2023-11-14T04:55:11.4858301Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2023-11-14T04:55:11.4859462Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2023-11-14T04:55:11.4861129Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2023-11-14T04:55:11.4862261Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2023-11-14T04:55:11.4863017Z + BUILD_DIR=build 2023-11-14T04:55:11.4863427Z + BUILD_RENAMED_DIR=build_renamed 2023-11-14T04:55:11.4863917Z + BUILD_BIN_DIR=build/bin 2023-11-14T04:55:11.4864337Z + export VALGRIND=ON 2023-11-14T04:55:11.4864736Z + VALGRIND=ON 2023-11-14T04:55:11.4865752Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2023-11-14T04:55:11.4866432Z + [[ 0 == \1 ]] 2023-11-14T04:55:11.4866785Z + [[ True == \1 ]] 2023-11-14T04:55:11.4867345Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2023-11-14T04:55:11.4867994Z ++ realpath build/custom_test_artifacts 2023-11-14T04:55:11.4875115Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2023-11-14T04:55:11.4878355Z ++ dirname .ci/pytorch/test.sh 2023-11-14T04:55:11.4886286Z + source .ci/pytorch/common.sh 2023-11-14T04:55:11.4890423Z +++ dirname .ci/pytorch/common.sh 2023-11-14T04:55:11.4898931Z ++ source .ci/pytorch/common_utils.sh 2023-11-14T04:55:11.4901018Z +++ declare -f -t trap_add 2023-11-14T04:55:11.4907840Z ++ set -ex 2023-11-14T04:55:11.4908370Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2023-11-14T04:55:11.4908975Z ++ BUILD_TEST_LIBTORCH=0 2023-11-14T04:55:11.4909476Z + echo 'Environment variables' 2023-11-14T04:55:11.4909934Z Environment variables 2023-11-14T04:55:11.4910333Z + env 2023-11-14T04:55:11.4916994Z INSTALLED_DB=yes 2023-11-14T04:55:11.4917576Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2023-11-14T04:55:11.4918077Z NVIDIA_VISIBLE_DEVICES=all 2023-11-14T04:55:11.4918563Z NV_NVML_DEV_VERSION=11.8.86-1 2023-11-14T04:55:11.4919524Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:55:11.4920261Z CONTINUE_THROUGH_ERROR=True 2023-11-14T04:55:11.4920876Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2023-11-14T04:55:11.4921566Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2023-11-14T04:55:11.4922258Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:55:11.4922882Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2023-11-14T04:55:11.4923423Z HOSTNAME=99f90b6e6ff3 2023-11-14T04:55:11.4924477Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.4925658Z GITHUB_ACTION=__self 2023-11-14T04:55:11.4926443Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2023-11-14T04:55:11.4935708Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 2023-11-14T04:55:11.4943591Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2023-11-14T04:55:11.4944222Z NV_NVTX_VERSION=11.8.86-1 2023-11-14T04:55:11.4944664Z GITHUB_RUN_NUMBER=155684 2023-11-14T04:55:11.4945096Z TEST_CONFIG=distributed 2023-11-14T04:55:11.4945521Z GITHUB_REPOSITORY_OWNER_ID=21003710 2023-11-14T04:55:11.4946117Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2023-11-14T04:55:11.4946697Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2023-11-14T04:55:11.4947230Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2023-11-14T04:55:11.4947758Z NV_LIBNPP_VERSION=11.8.0.86-1 2023-11-14T04:55:11.4948238Z GITHUB_TRIGGERING_ACTOR=clee2000 2023-11-14T04:55:11.4948792Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2023-11-14T04:55:11.4949389Z GITHUB_REF_TYPE=branch 2023-11-14T04:55:11.4949817Z TORCH_CUDA_ARCH_LIST=Maxwell 2023-11-14T04:55:11.4950272Z NCCL_VERSION=2.16.2-1 2023-11-14T04:55:11.4950765Z BASE_SHA=e1c872e009e8aa07281f38723353dc43baf2fc12 2023-11-14T04:55:11.4951318Z XLA_CUDA= 2023-11-14T04:55:11.4951647Z HUGGING_FACE_HUB_TOKEN= 2023-11-14T04:55:11.4952157Z *** 2023-11-14T04:55:11.4952526Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2023-11-14T04:55:11.4953003Z GITHUB_REPOSITORY_ID=65600975 2023-11-14T04:55:11.4953458Z GITHUB_ACTIONS=true 2023-11-14T04:55:11.4953874Z NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:55:11.4954467Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2023-11-14T04:55:11.4955143Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2023-11-14T04:55:11.4955728Z SHA1=fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:55:11.4956345Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2023-11-14T04:55:11.4956958Z GITHUB_SHA=35b9c00c6974fbcfd0eb53491c53c61a6c439831 2023-11-14T04:55:11.4957856Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/112098/merge 2023-11-14T04:55:11.4958647Z UCC_HOME=/usr 2023-11-14T04:55:11.4959078Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2023-11-14T04:55:11.4959636Z NVIDIA_PRODUCT_NAME=CUDA 2023-11-14T04:55:11.4960228Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2023-11-14T04:55:11.4960862Z GITHUB_REF=refs/pull/112098/merge 2023-11-14T04:55:11.4961588Z NV_CUDA_CUDART_VERSION=11.8.89-1 2023-11-14T04:55:11.4962066Z SHARD_NUMBER=1 2023-11-14T04:55:11.4962471Z GITHUB_REF_PROTECTED=false 2023-11-14T04:55:11.4962941Z HOME=/var/lib/jenkins 2023-11-14T04:55:11.4963413Z GITHUB_API_URL=https://api.github.com 2023-11-14T04:55:11.4964001Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2023-11-14T04:55:11.4964637Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2023-11-14T04:55:11.4965248Z SCCACHE_S3_KEY_PREFIX=pull 2023-11-14T04:55:11.4965713Z CUDA_VERSION=11.8.0 2023-11-14T04:55:11.4966285Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2023-11-14T04:55:11.4966846Z NUM_TEST_SHARDS=3 2023-11-14T04:55:11.4967241Z UCX_HOME=/usr 2023-11-14T04:55:11.4967924Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2023-11-14T04:55:11.4969344Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.4970471Z PYTORCH_RETRY_TEST_CASES=1 2023-11-14T04:55:11.4972055Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.4973503Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2023-11-14T04:55:11.4974337Z GITHUB_EVENT_NAME=pull_request 2023-11-14T04:55:11.4974802Z DASHBOARD_TAG= 2023-11-14T04:55:11.4975179Z GITHUB_RUN_ID=6859165800 2023-11-14T04:55:11.4975731Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2023-11-14T04:55:11.4976407Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2023-11-14T04:55:11.4977672Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.4978758Z GITHUB_ACTOR=clee2000 2023-11-14T04:55:11.4979223Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2023-11-14T04:55:11.4979832Z PR_NUMBER=112098 2023-11-14T04:55:11.4980227Z GITHUB_RUN_ATTEMPT=1 2023-11-14T04:55:11.4980634Z VALGRIND=ON 2023-11-14T04:55:11.4981007Z ANACONDA_PYTHON_VERSION=3.10 2023-11-14T04:55:11.4981556Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2023-11-14T04:55:11.4982138Z TERM=xterm 2023-11-14T04:55:11.4982557Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2023-11-14T04:55:11.4983054Z INSTALLED_VISION=yes 2023-11-14T04:55:11.4983461Z BRANCH=pull/112098 2023-11-14T04:55:11.4983865Z OPENSSL_ROOT_DIR=/opt/openssl 2023-11-14T04:55:11.4984347Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2023-11-14T04:55:11.4984878Z CUDA_PATH=/usr/local/cuda 2023-11-14T04:55:11.4985827Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2023-11-14T04:55:11.4986769Z GITHUB_SERVER_URL=https://github.com 2023-11-14T04:55:11.4987389Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2023-11-14T04:55:11.4987983Z REENABLED_ISSUES= 2023-11-14T04:55:11.4988334Z SHLVL=1 2023-11-14T04:55:11.4988668Z MAX_JOBS=30 2023-11-14T04:55:11.4989062Z NV_CUDA_LIB_VERSION=11.8.0-1 2023-11-14T04:55:11.4989491Z NVARCH=x86_64 2023-11-14T04:55:11.4989864Z GITHUB_ACTOR_ID=44682903 2023-11-14T04:55:11.4990447Z GITHUB_WORKFLOW_SHA=35b9c00c6974fbcfd0eb53491c53c61a6c439831 2023-11-14T04:55:11.4991081Z GITHUB_REF_NAME=112098/merge 2023-11-14T04:55:11.4991625Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2023-11-14T04:55:11.4992463Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2023-11-14T04:55:11.4993172Z GITHUB_JOB=test 2023-11-14T04:55:11.4993670Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2023-11-14T04:55:11.4994374Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-11-14T04:55:11.4995065Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2023-11-14T04:55:11.4995620Z GITHUB_REPOSITORY=pytorch/pytorch 2023-11-14T04:55:11.4996141Z NV_NVPROF_VERSION=11.8.87-1 2023-11-14T04:55:11.4996578Z GITHUB_RETENTION_DAYS=90 2023-11-14T04:55:11.4997018Z OPENSSL_DIR=/opt/openssl 2023-11-14T04:55:11.4997465Z GITHUB_ACTION_REPOSITORY= 2023-11-14T04:55:11.4998787Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-11-14T04:55:11.5000331Z GITHUB_BASE_REF=main 2023-11-14T04:55:11.5000784Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2023-11-14T04:55:11.5001232Z CI=true 2023-11-14T04:55:11.5001629Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2023-11-14T04:55:11.5002156Z GITHUB_REPOSITORY_OWNER=pytorch 2023-11-14T04:55:11.5002608Z JOB_ID=18651430019 2023-11-14T04:55:11.5002999Z INSTALLED_PROTOBUF=yes 2023-11-14T04:55:11.5003424Z GITHUB_HEAD_REF=csl/test3 2023-11-14T04:55:11.5003835Z GITHUB_ACTION_REF= 2023-11-14T04:55:11.5004382Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2023-11-14T04:55:11.5004975Z GITHUB_WORKFLOW=pull 2023-11-14T04:55:11.5005386Z DEBIAN_FRONTEND=noninteractive 2023-11-14T04:55:11.5006544Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ee07772e-fbb7-4563-9755-6baa0abbdfdb 2023-11-14T04:55:11.5007630Z SKIP_SCCACHE_INITIALIZATION=1 2023-11-14T04:55:11.5008067Z _=/usr/bin/env 2023-11-14T04:55:11.5008492Z + echo 'Testing pytorch' 2023-11-14T04:55:11.5008934Z Testing pytorch 2023-11-14T04:55:11.5009325Z + export LANG=C.UTF-8 2023-11-14T04:55:11.5009753Z + LANG=C.UTF-8 2023-11-14T04:55:11.5010125Z + PR_NUMBER=112098 2023-11-14T04:55:11.5010534Z + [[ distributed == \d\e\f\a\u\l\t ]] 2023-11-14T04:55:11.5011392Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2023-11-14T04:55:11.5012104Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2023-11-14T04:55:11.5012691Z + [[ distributed == \s\l\o\w ]] 2023-11-14T04:55:11.5013389Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2023-11-14T04:55:11.5014194Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2023-11-14T04:55:11.5014835Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2023-11-14T04:55:11.5015433Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2023-11-14T04:55:11.5015967Z + [[ distributed == *crossref* ]] 2023-11-14T04:55:11.5016699Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2023-11-14T04:55:11.5017483Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2023-11-14T04:55:11.5018162Z + pip_install --user ninja==1.10.2 2023-11-14T04:55:11.5018841Z + pip install --progress-bar off --user ninja==1.10.2 2023-11-14T04:55:12.0635253Z Collecting ninja==1.10.2 2023-11-14T04:55:12.1103050Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2023-11-14T04:55:13.1778811Z Installing collected packages: ninja 2023-11-14T04:55:13.1880490Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2023-11-14T04:55:13.1882113Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2023-11-14T04:55:13.1938367Z Successfully installed ninja-1.10.2 2023-11-14T04:55:13.2644360Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-11-14T04:55:13.2647158Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-11-14T04:55:13.2648989Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2023-11-14T04:55:13.2649748Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2023-11-14T04:55:13.2650494Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2023-11-14T04:55:13.2652109Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2023-11-14T04:55:13.2653482Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2023-11-14T04:55:13.2654324Z + cd test 2023-11-14T04:55:13.2654985Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2023-11-14T04:55:14.8587723Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2023-11-14T04:55:14.8588450Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2023-11-14T04:55:14.8589222Z + DYNAMO_BENCHMARK_FLAGS=() 2023-11-14T04:55:14.8589715Z + [[ distributed == *dynamo_eager* ]] 2023-11-14T04:55:14.8590623Z + [[ distributed == *aot_eager* ]] 2023-11-14T04:55:14.8591117Z + [[ distributed == *aot_inductor* ]] 2023-11-14T04:55:14.8591636Z + [[ distributed == *inductor* ]] 2023-11-14T04:55:14.8592133Z + [[ distributed == *dynamic* ]] 2023-11-14T04:55:14.8592660Z + [[ distributed == *cpu_accuracy* ]] 2023-11-14T04:55:14.8593351Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2023-11-14T04:55:14.8602764Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *tbb* ]] 2023-11-14T04:55:14.8619569Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2023-11-14T04:55:14.8620348Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2023-11-14T04:55:14.8622483Z + cd test 2023-11-14T04:55:14.8623065Z + python -c 'import torch; print(torch.__config__.show())' 2023-11-14T04:55:16.2736038Z PyTorch built with: 2023-11-14T04:55:16.2736650Z - GCC 9.4 2023-11-14T04:55:16.2737065Z - C++ Version: 201703 2023-11-14T04:55:16.2738160Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2023-11-14T04:55:16.2739562Z - Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4) 2023-11-14T04:55:16.2740402Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2023-11-14T04:55:16.2741026Z - LAPACK is enabled (usually provided by MKL) 2023-11-14T04:55:16.2741628Z - NNPACK is enabled 2023-11-14T04:55:16.2742097Z - CPU capability usage: AVX2 2023-11-14T04:55:16.2742583Z - CUDA Runtime 11.8 2023-11-14T04:55:16.2743228Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2023-11-14T04:55:16.2743919Z - CuDNN 8.7 2023-11-14T04:55:16.2744301Z - Magma 2.6.1 2023-11-14T04:55:16.2753667Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.2.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 2023-11-14T04:55:16.2762193Z 2023-11-14T04:55:16.4955265Z + cd test 2023-11-14T04:55:16.4956096Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2023-11-14T04:55:17.8597965Z ATen/Parallel: 2023-11-14T04:55:17.8598472Z at::get_num_threads() : 16 2023-11-14T04:55:17.8598966Z at::get_num_interop_threads() : 32 2023-11-14T04:55:17.8599506Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2023-11-14T04:55:17.8600004Z omp_get_max_threads() : 16 2023-11-14T04:55:17.8601175Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2023-11-14T04:55:17.8602203Z mkl_get_max_threads() : 16 2023-11-14T04:55:17.8602992Z Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4) 2023-11-14T04:55:17.8603772Z std::thread::hardware_concurrency() : 32 2023-11-14T04:55:17.8604309Z Environment variables: 2023-11-14T04:55:17.8604768Z OMP_NUM_THREADS : [not set] 2023-11-14T04:55:17.8605209Z MKL_NUM_THREADS : [not set] 2023-11-14T04:55:17.8605675Z ATen parallel backend: OpenMP 2023-11-14T04:55:17.8605995Z 2023-11-14T04:55:18.0605471Z + [[ distributed == *backward* ]] 2023-11-14T04:55:18.0606050Z + [[ distributed == *xla* ]] 2023-11-14T04:55:18.0606575Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2023-11-14T04:55:18.0607408Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2023-11-14T04:55:18.0608063Z + [[ distributed == distributed ]] 2023-11-14T04:55:18.0608537Z + test_distributed 2023-11-14T04:55:18.0609036Z + echo 'Testing distributed python tests' 2023-11-14T04:55:18.0609608Z Testing distributed python tests 2023-11-14T04:55:18.0610359Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2023-11-14T04:55:18.1759535Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2023-11-14T04:55:18.1760979Z import pkg_resources 2023-11-14T04:55:19.8713382Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2023-11-14T04:55:19.9462955Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2023-11-14T04:55:20.0250999Z Ignoring disabled issues: [''] 2023-11-14T04:55:20.0258593Z Received 182 tests to prioritize 2023-11-14T04:55:20.0259709Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:20.0261286Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:20.0262964Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:20.0264744Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:20.0266187Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:20.0267617Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:20.0268464Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:20.0269143Z distributed/_composable/test_checkpoint 2023-11-14T04:55:20.0269730Z distributed/_composable/test_compose 2023-11-14T04:55:20.0270276Z distributed/_composable/test_contract 2023-11-14T04:55:20.0270824Z distributed/_composable/test_replicate 2023-11-14T04:55:20.0271454Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:20.0272170Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:20.0272872Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:20.0273605Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:20.0274311Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:20.0274977Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:20.0275661Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:20.0276341Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:20.0277092Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:20.0277857Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:20.0278875Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:20.0280057Z distributed/_shard/test_sharder 2023-11-14T04:55:20.0281011Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:20.0282047Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:20.0282910Z distributed/_spmd/test_tracing 2023-11-14T04:55:20.0283436Z distributed/_spmd/test_transformation 2023-11-14T04:55:20.0284258Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:20.0285144Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:20.0285813Z distributed/_tensor/test_api 2023-11-14T04:55:20.0286500Z distributed/_tensor/test_common_rules 2023-11-14T04:55:20.0287343Z distributed/_tensor/test_dtensor 2023-11-14T04:55:20.0288328Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:20.0289216Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:20.0290270Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:20.0291310Z distributed/_tensor/test_init 2023-11-14T04:55:20.0292039Z distributed/_tensor/test_math_ops 2023-11-14T04:55:20.0292644Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:20.0293606Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:20.0294372Z distributed/_tensor/test_optimizers 2023-11-14T04:55:20.0295227Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:20.0296270Z distributed/_tensor/test_random_ops 2023-11-14T04:55:20.0297005Z distributed/_tensor/test_redistribute 2023-11-14T04:55:20.0297808Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:20.0298642Z distributed/_tensor/test_utils 2023-11-14T04:55:20.0299353Z distributed/_tensor/test_view_ops 2023-11-14T04:55:20.0300333Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:20.0301221Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:20.0302269Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:20.0303295Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:20.0304371Z distributed/algorithms/test_join 2023-11-14T04:55:20.0305225Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:20.0306111Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:20.0306737Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:20.0307640Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:20.0308598Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:20.0309308Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:20.0310191Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:20.0311067Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:20.0311904Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:20.0313063Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:20.0314123Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:20.0314881Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:20.0315876Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:20.0316682Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:20.0317757Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:20.0318356Z distributed/checkpoint/test_planner 2023-11-14T04:55:20.0319161Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:20.0320186Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:20.0321085Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:20.0321653Z distributed/checkpoint/test_traverse 2023-11-14T04:55:20.0322677Z distributed/checkpoint/test_utils 2023-11-14T04:55:20.0323226Z distributed/elastic/events/lib_test 2023-11-14T04:55:20.0323746Z distributed/elastic/metrics/api_test 2023-11-14T04:55:20.0324631Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:20.0325465Z distributed/elastic/timer/api_test 2023-11-14T04:55:20.0326044Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:20.0326824Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:20.0327563Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:20.0328141Z distributed/elastic/utils/logging_test 2023-11-14T04:55:20.0328987Z distributed/elastic/utils/util_test 2023-11-14T04:55:20.0329562Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:20.0330450Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:20.0331251Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:20.0332072Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:20.0332932Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:20.0333528Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:20.0334304Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:20.0335114Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:20.0335693Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:20.0336479Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:20.0337067Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:20.0337905Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:20.0338684Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:20.0339537Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:20.0340124Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:20.0340641Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:20.0341433Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:20.0342034Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:20.0342605Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:20.0343100Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:20.0343620Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:20.0344122Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:20.0344644Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:20.0345259Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:20.0345896Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:20.0346482Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:20.0347037Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:20.0347584Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:20.0348156Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:20.0348764Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:20.0349333Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:20.0349891Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:20.0350432Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:20.0350979Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:20.0351743Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:20.0352371Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:20.0352957Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:20.0353446Z distributed/fsdp/test_utils 2023-11-14T04:55:20.0354061Z distributed/fsdp/test_wrap 2023-11-14T04:55:20.0354575Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:20.0355181Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:20.0355826Z distributed/optim/test_named_optimizer 2023-11-14T04:55:20.0356438Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:20.0357076Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:20.0357645Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:20.0358333Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:20.0359014Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:20.0359602Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:20.0360235Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:20.0360869Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:20.0361529Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:20.0362186Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:20.0362749Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:20.0363296Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:20.0363876Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:20.0364503Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:20.0365145Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:20.0365734Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:20.0366313Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:20.0366874Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:20.0367422Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:20.0367977Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:20.0368526Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:20.0369113Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:20.0369703Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:20.0370268Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:20.0371071Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:20.0371606Z distributed/rpc/test_share_memory 2023-11-14T04:55:20.0372120Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:20.0372731Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:20.0373580Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:20.0374251Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:20.0374920Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:20.0375569Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:20.0376190Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:20.0376885Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:20.0377541Z distributed/test_c10d_common 2023-11-14T04:55:20.0378030Z distributed/test_c10d_functional_native 2023-11-14T04:55:20.0378576Z distributed/test_c10d_gloo 2023-11-14T04:55:20.0379035Z distributed/test_c10d_logger 2023-11-14T04:55:20.0379485Z distributed/test_c10d_nccl 2023-11-14T04:55:20.0379986Z distributed/test_c10d_object_collectives 2023-11-14T04:55:20.0380536Z distributed/test_c10d_pypg 2023-11-14T04:55:20.0380988Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:20.0381506Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:20.0382010Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:20.0382483Z distributed/test_c10d_ucc 2023-11-14T04:55:20.0382952Z distributed/test_collective_utils 2023-11-14T04:55:20.0383500Z distributed/test_compute_comm_reordering 2023-11-14T04:55:20.0384040Z distributed/test_data_parallel 2023-11-14T04:55:20.0384530Z distributed/test_device_mesh 2023-11-14T04:55:20.0385019Z distributed/test_distributed_spawn 2023-11-14T04:55:20.0385538Z distributed/test_dynamo_distributed 2023-11-14T04:55:20.0386055Z distributed/test_fake_pg 2023-11-14T04:55:20.0386517Z distributed/test_functional_api 2023-11-14T04:55:20.0387017Z distributed/test_inductor_collectives 2023-11-14T04:55:20.0387545Z distributed/test_launcher 2023-11-14T04:55:20.0388155Z distributed/test_multi_threaded_pg 2023-11-14T04:55:20.0388673Z distributed/test_nccl 2023-11-14T04:55:20.0389111Z distributed/test_pg_wrapper 2023-11-14T04:55:20.0389578Z distributed/test_store 2023-11-14T04:55:20.0390248Z Heuristic PreviouslyFailedInPR identified 1 tests to prioritize (0.55%%) 2023-11-14T04:55:20.0391018Z High Relevance tests (1): 2023-11-14T04:55:20.0391484Z distributed/test_distributed_spawn 2023-11-14T04:55:20.0391992Z Unranked Relevance tests (181): 2023-11-14T04:55:20.0392634Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:20.0393504Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:20.0394351Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:20.0395268Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:20.0396168Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:20.0397040Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:20.0397828Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:20.0398484Z distributed/_composable/test_checkpoint 2023-11-14T04:55:20.0399058Z distributed/_composable/test_compose 2023-11-14T04:55:20.0399602Z distributed/_composable/test_contract 2023-11-14T04:55:20.0400158Z distributed/_composable/test_replicate 2023-11-14T04:55:20.0400771Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:20.0401482Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:20.0402192Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:20.0402912Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:20.0403618Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:20.0404304Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:20.0404960Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:20.0405635Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:20.0406398Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:20.0407138Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:20.0407970Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:20.0408590Z distributed/_shard/test_sharder 2023-11-14T04:55:20.0409088Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:20.0409628Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:20.0410149Z distributed/_spmd/test_tracing 2023-11-14T04:55:20.0410640Z distributed/_spmd/test_transformation 2023-11-14T04:55:20.0411436Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:20.0412094Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:20.0412692Z distributed/_tensor/test_api 2023-11-14T04:55:20.0413185Z distributed/_tensor/test_common_rules 2023-11-14T04:55:20.0413722Z distributed/_tensor/test_dtensor 2023-11-14T04:55:20.0414242Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:20.0414811Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:20.0415363Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:20.0415895Z distributed/_tensor/test_init 2023-11-14T04:55:20.0416392Z distributed/_tensor/test_math_ops 2023-11-14T04:55:20.0416915Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:20.0417432Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:20.0417970Z distributed/_tensor/test_optimizers 2023-11-14T04:55:20.0418512Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:20.0419046Z distributed/_tensor/test_random_ops 2023-11-14T04:55:20.0419578Z distributed/_tensor/test_redistribute 2023-11-14T04:55:20.0420121Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:20.0420624Z distributed/_tensor/test_utils 2023-11-14T04:55:20.0421123Z distributed/_tensor/test_view_ops 2023-11-14T04:55:20.0421666Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:20.0422416Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:20.0423093Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:20.0423826Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:20.0424457Z distributed/algorithms/test_join 2023-11-14T04:55:20.0425040Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:20.0425976Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:20.0426573Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:20.0427158Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:20.0427741Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:20.0428367Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:20.0428998Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:20.0429673Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:20.0430400Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:20.0431074Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:20.0431712Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:20.0432424Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:20.0433087Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:20.0433662Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:20.0434260Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:20.0434801Z distributed/checkpoint/test_planner 2023-11-14T04:55:20.0435348Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:20.0435942Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:20.0436536Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:20.0437114Z distributed/checkpoint/test_traverse 2023-11-14T04:55:20.0437653Z distributed/checkpoint/test_utils 2023-11-14T04:55:20.0438161Z distributed/elastic/events/lib_test 2023-11-14T04:55:20.0438695Z distributed/elastic/metrics/api_test 2023-11-14T04:55:20.0439280Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:20.0439858Z distributed/elastic/timer/api_test 2023-11-14T04:55:20.0440440Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:20.0441067Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:20.0441823Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:20.0442414Z distributed/elastic/utils/logging_test 2023-11-14T04:55:20.0442968Z distributed/elastic/utils/util_test 2023-11-14T04:55:20.0443506Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:20.0444122Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:20.0444703Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:20.0445243Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:20.0445829Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:20.0446391Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:20.0446939Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:20.0447449Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:20.0447992Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:20.0448534Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:20.0449129Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:20.0449686Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:20.0450230Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:20.0451228Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:20.0451814Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:20.0452300Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:20.0452847Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:20.0453436Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:20.0453991Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:20.0454503Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:20.0455019Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:20.0455498Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:20.0456037Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:20.0456775Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:20.0457406Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:20.0458011Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:20.0458567Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:20.0459086Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:20.0459673Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:20.0460275Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:20.0460846Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:20.0461404Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:20.0461937Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:20.0462481Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:20.0463068Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:20.0463688Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:20.0464273Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:20.0464763Z distributed/fsdp/test_utils 2023-11-14T04:55:20.0465238Z distributed/fsdp/test_wrap 2023-11-14T04:55:20.0465723Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:20.0466323Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:20.0466968Z distributed/optim/test_named_optimizer 2023-11-14T04:55:20.0467582Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:20.0468201Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:20.0468782Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:20.0469462Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:20.0470125Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:20.0470732Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:20.0471361Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:20.0471979Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:20.0472653Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:20.0473308Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:20.0473858Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:20.0474420Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:20.0474995Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:20.0475743Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:20.0476399Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:20.0476988Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:20.0477547Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:20.0478123Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:20.0478670Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:20.0479211Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:20.0479777Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:20.0480364Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:20.0480936Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:20.0481521Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:20.0482104Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:20.0482611Z distributed/rpc/test_share_memory 2023-11-14T04:55:20.0483144Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:20.0483762Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:20.0484423Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:20.0485109Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:20.0485774Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:20.0486406Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:20.0487047Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:20.0487742Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:20.0488396Z distributed/test_c10d_common 2023-11-14T04:55:20.0488880Z distributed/test_c10d_functional_native 2023-11-14T04:55:20.0489419Z distributed/test_c10d_gloo 2023-11-14T04:55:20.0489885Z distributed/test_c10d_logger 2023-11-14T04:55:20.0490429Z distributed/test_c10d_nccl 2023-11-14T04:55:20.0491134Z distributed/test_c10d_object_collectives 2023-11-14T04:55:20.0491692Z distributed/test_c10d_pypg 2023-11-14T04:55:20.0492143Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:20.0492658Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:20.0493164Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:20.0493640Z distributed/test_c10d_ucc 2023-11-14T04:55:20.0494112Z distributed/test_collective_utils 2023-11-14T04:55:20.0494657Z distributed/test_compute_comm_reordering 2023-11-14T04:55:20.0495198Z distributed/test_data_parallel 2023-11-14T04:55:20.0495688Z distributed/test_device_mesh 2023-11-14T04:55:20.0496179Z distributed/test_dynamo_distributed 2023-11-14T04:55:20.0496681Z distributed/test_fake_pg 2023-11-14T04:55:20.0497139Z distributed/test_functional_api 2023-11-14T04:55:20.0497656Z distributed/test_inductor_collectives 2023-11-14T04:55:20.0498168Z distributed/test_launcher 2023-11-14T04:55:20.0498644Z distributed/test_multi_threaded_pg 2023-11-14T04:55:20.0499154Z distributed/test_nccl 2023-11-14T04:55:20.0499574Z distributed/test_pg_wrapper 2023-11-14T04:55:20.0500035Z distributed/test_store 2023-11-14T04:55:20.0500675Z Heuristic EditedByPR identified 0 tests to prioritize (0.00%%) 2023-11-14T04:55:20.2621499Z Heuristic CorrelatedWithHistoricalFailures identified 49 tests to prioritize (26.92%%) 2023-11-14T04:55:20.2622564Z Probable Relevance tests (49): 2023-11-14T04:55:20.2623076Z distributed/test_distributed_spawn 2023-11-14T04:55:20.2623642Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:20.2624173Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:20.2624704Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:20.2625525Z distributed/test_multi_threaded_pg 2023-11-14T04:55:20.2626292Z distributed/test_c10d_nccl 2023-11-14T04:55:20.2627138Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:20.2628332Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:20.2628902Z distributed/_tensor/test_dtensor 2023-11-14T04:55:20.2629404Z distributed/test_c10d_ucc 2023-11-14T04:55:20.2629868Z distributed/test_pg_wrapper 2023-11-14T04:55:20.2630349Z distributed/test_dynamo_distributed 2023-11-14T04:55:20.2631549Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:20.2632373Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:20.2633386Z distributed/test_c10d_gloo 2023-11-14T04:55:20.2634260Z distributed/test_launcher 2023-11-14T04:55:20.2634908Z distributed/test_inductor_collectives 2023-11-14T04:55:20.2635795Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:20.2636693Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:20.2637551Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:20.2638377Z distributed/test_c10d_common 2023-11-14T04:55:20.2639007Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:20.2640097Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:20.2640988Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:20.2641671Z distributed/test_c10d_object_collectives 2023-11-14T04:55:20.2642431Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:20.2643325Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:20.2644467Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:20.2645309Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:20.2646061Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:20.2646668Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:20.2647517Z distributed/test_data_parallel 2023-11-14T04:55:20.2648067Z distributed/_spmd/test_tracing 2023-11-14T04:55:20.2649093Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:20.2650138Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:20.2651600Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:20.2652284Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:20.2653452Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:20.2654300Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:20.2654927Z distributed/test_compute_comm_reordering 2023-11-14T04:55:20.2655655Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:20.2656503Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:20.2657489Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:20.2658227Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:20.2658776Z distributed/_tensor/test_utils 2023-11-14T04:55:20.2659751Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:20.2660814Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:20.2661487Z distributed/checkpoint/test_planner 2023-11-14T04:55:20.2662141Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:20.2662775Z Unranked Relevance tests (133): 2023-11-14T04:55:20.2663468Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:20.2664400Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:20.2665225Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:20.2665918Z distributed/_composable/test_checkpoint 2023-11-14T04:55:20.2666484Z distributed/_composable/test_compose 2023-11-14T04:55:20.2667017Z distributed/_composable/test_contract 2023-11-14T04:55:20.2667582Z distributed/_composable/test_replicate 2023-11-14T04:55:20.2668226Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:20.2668924Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:20.2669663Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:20.2670372Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:20.2671046Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:20.2671728Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:20.2672405Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:20.2673153Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:20.2673913Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:20.2674821Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:20.2675445Z distributed/_shard/test_sharder 2023-11-14T04:55:20.2675943Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:20.2676487Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:20.2677030Z distributed/_spmd/test_transformation 2023-11-14T04:55:20.2677637Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:20.2678255Z distributed/_tensor/test_api 2023-11-14T04:55:20.2678751Z distributed/_tensor/test_common_rules 2023-11-14T04:55:20.2679291Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:20.2679831Z distributed/_tensor/test_init 2023-11-14T04:55:20.2680326Z distributed/_tensor/test_math_ops 2023-11-14T04:55:20.2680837Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:20.2681381Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:20.2681931Z distributed/_tensor/test_optimizers 2023-11-14T04:55:20.2682449Z distributed/_tensor/test_random_ops 2023-11-14T04:55:20.2682997Z distributed/_tensor/test_redistribute 2023-11-14T04:55:20.2683544Z distributed/_tensor/test_view_ops 2023-11-14T04:55:20.2684068Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:20.2684647Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:20.2685290Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:20.2686007Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:20.2686651Z distributed/algorithms/test_join 2023-11-14T04:55:20.2687230Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:20.2687871Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:20.2688477Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:20.2689062Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:20.2689714Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:20.2690360Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:20.2691292Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:20.2691929Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:20.2692561Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:20.2693279Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:20.2693938Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:20.2694517Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:20.2695121Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:20.2695704Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:20.2696317Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:20.2696900Z distributed/checkpoint/test_traverse 2023-11-14T04:55:20.2697426Z distributed/checkpoint/test_utils 2023-11-14T04:55:20.2697961Z distributed/elastic/events/lib_test 2023-11-14T04:55:20.2698512Z distributed/elastic/metrics/api_test 2023-11-14T04:55:20.2699086Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:20.2699682Z distributed/elastic/timer/api_test 2023-11-14T04:55:20.2700267Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:20.2700876Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:20.2701469Z distributed/elastic/utils/logging_test 2023-11-14T04:55:20.2702024Z distributed/elastic/utils/util_test 2023-11-14T04:55:20.2702582Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:20.2703170Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:20.2703735Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:20.2704308Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:20.2704881Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:20.2705446Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:20.2705942Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:20.2706536Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:20.2707130Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:20.2707716Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:20.2708449Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:20.2709000Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:20.2709514Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:20.2709996Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:20.2710546Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:20.2711173Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:20.2711743Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:20.2712280Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:20.2712845Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:20.2713406Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:20.2713944Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:20.2714468Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:20.2714968Z distributed/fsdp/test_utils 2023-11-14T04:55:20.2715449Z distributed/fsdp/test_wrap 2023-11-14T04:55:20.2715940Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:20.2716541Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:20.2717190Z distributed/optim/test_named_optimizer 2023-11-14T04:55:20.2717805Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:20.2718425Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:20.2719011Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:20.2719701Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:20.2720362Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:20.2720972Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:20.2721609Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:20.2722224Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:20.2722956Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:20.2723725Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:20.2724291Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:20.2724858Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:20.2725449Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:20.2726050Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:20.2726704Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:20.2727292Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:20.2727853Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:20.2728462Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:20.2729016Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:20.2729552Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:20.2730119Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:20.2730892Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:20.2731510Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:20.2732039Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:20.2732570Z distributed/rpc/test_share_memory 2023-11-14T04:55:20.2733162Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:20.2733828Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:20.2734501Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:20.2735150Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:20.2735772Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:20.2736468Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:20.2737153Z distributed/test_c10d_functional_native 2023-11-14T04:55:20.2737678Z distributed/test_c10d_logger 2023-11-14T04:55:20.2738154Z distributed/test_c10d_pypg 2023-11-14T04:55:20.2738627Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:20.2739119Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:20.2739622Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:20.2740127Z distributed/test_collective_utils 2023-11-14T04:55:20.2740647Z distributed/test_device_mesh 2023-11-14T04:55:20.2741123Z distributed/test_fake_pg 2023-11-14T04:55:20.2741559Z distributed/test_functional_api 2023-11-14T04:55:20.2742042Z distributed/test_nccl 2023-11-14T04:55:20.2742616Z distributed/test_store 2023-11-14T04:55:20.3293243Z Heuristic HistorialEditedFiles identified 63 tests to prioritize (34.62%%) 2023-11-14T04:55:20.3294486Z Probable Relevance tests (63): 2023-11-14T04:55:20.3294978Z distributed/test_c10d_ucc 2023-11-14T04:55:20.3295653Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:20.3296423Z distributed/test_c10d_nccl 2023-11-14T04:55:20.3297014Z distributed/test_data_parallel 2023-11-14T04:55:20.3297754Z distributed/test_nccl 2023-11-14T04:55:20.3298243Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:20.3298806Z distributed/test_store 2023-11-14T04:55:20.3299392Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:20.3300081Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:20.3300729Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:20.3301314Z distributed/test_c10d_gloo 2023-11-14T04:55:20.3301780Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:20.3302286Z distributed/test_c10d_common 2023-11-14T04:55:20.3302760Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:20.3303401Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:20.3304111Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:20.3304768Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:20.3305405Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:20.3305988Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:20.3306465Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:20.3306980Z distributed/elastic/events/lib_test 2023-11-14T04:55:20.3307527Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:20.3308110Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:20.3308947Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:20.3309660Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:20.3310265Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:20.3310886Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:20.3311518Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:20.3312193Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:20.3312833Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:20.3313699Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:20.3314483Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:20.3315552Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:20.3316274Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:20.3316927Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:20.3317972Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:20.3318539Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:20.3319121Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:20.3319666Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:20.3320198Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:20.3320771Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:20.3321357Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:20.3321929Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:20.3322528Z distributed/test_dynamo_distributed 2023-11-14T04:55:20.3323068Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:20.3323611Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:20.3324158Z distributed/_tensor/test_view_ops 2023-11-14T04:55:20.3324665Z distributed/test_launcher 2023-11-14T04:55:20.3325134Z distributed/_composable/test_replicate 2023-11-14T04:55:20.3325763Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:20.3326386Z distributed/_tensor/test_dtensor 2023-11-14T04:55:20.3326970Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:20.3327604Z distributed/algorithms/test_join 2023-11-14T04:55:20.3328150Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:20.3328763Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:20.3329667Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:20.3330341Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:20.3331186Z distributed/rpc/test_share_memory 2023-11-14T04:55:20.3331784Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:20.3332448Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:20.3333033Z distributed/test_c10d_object_collectives 2023-11-14T04:55:20.3333575Z distributed/test_c10d_pypg 2023-11-14T04:55:20.3334258Z distributed/test_pg_wrapper 2023-11-14T04:55:20.3335011Z Unranked Relevance tests (119): 2023-11-14T04:55:20.3335738Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:20.3336777Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:20.3338078Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:20.3339108Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:20.3340034Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:20.3340895Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:20.3341666Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:20.3342339Z distributed/_composable/test_checkpoint 2023-11-14T04:55:20.3342892Z distributed/_composable/test_compose 2023-11-14T04:55:20.3343418Z distributed/_composable/test_contract 2023-11-14T04:55:20.3344049Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:20.3344757Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:20.3345474Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:20.3346178Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:20.3346987Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:20.3347670Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:20.3348383Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:20.3349146Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:20.3349743Z distributed/_shard/test_sharder 2023-11-14T04:55:20.3350250Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:20.3350786Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:20.3351292Z distributed/_spmd/test_tracing 2023-11-14T04:55:20.3351799Z distributed/_spmd/test_transformation 2023-11-14T04:55:20.3352384Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:20.3353015Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:20.3353621Z distributed/_tensor/test_api 2023-11-14T04:55:20.3354109Z distributed/_tensor/test_common_rules 2023-11-14T04:55:20.3354652Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:20.3355220Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:20.3355772Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:20.3356291Z distributed/_tensor/test_init 2023-11-14T04:55:20.3356788Z distributed/_tensor/test_math_ops 2023-11-14T04:55:20.3357311Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:20.3357829Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:20.3358364Z distributed/_tensor/test_optimizers 2023-11-14T04:55:20.3358907Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:20.3359440Z distributed/_tensor/test_random_ops 2023-11-14T04:55:20.3359971Z distributed/_tensor/test_redistribute 2023-11-14T04:55:20.3360514Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:20.3361015Z distributed/_tensor/test_utils 2023-11-14T04:55:20.3361529Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:20.3362099Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:20.3362690Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:20.3363352Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:20.3363956Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:20.3364534Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:20.3365317Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:20.3365966Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:20.3366608Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:20.3367215Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:20.3367916Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:20.3368631Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:20.3369204Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:20.3369782Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:20.3370339Z distributed/checkpoint/test_planner 2023-11-14T04:55:20.3371158Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:20.3371736Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:20.3372350Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:20.3372928Z distributed/checkpoint/test_traverse 2023-11-14T04:55:20.3373447Z distributed/checkpoint/test_utils 2023-11-14T04:55:20.3373986Z distributed/elastic/metrics/api_test 2023-11-14T04:55:20.3374519Z distributed/elastic/timer/api_test 2023-11-14T04:55:20.3375065Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:20.3375676Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:20.3376262Z distributed/elastic/utils/logging_test 2023-11-14T04:55:20.3376796Z distributed/elastic/utils/util_test 2023-11-14T04:55:20.3377370Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:20.3377950Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:20.3378491Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:20.3379078Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:20.3379643Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:20.3380300Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:20.3380828Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:20.3381376Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:20.3381957Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:20.3382503Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:20.3383062Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:20.3383654Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:20.3384221Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:20.3384721Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:20.3385250Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:20.3385836Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:20.3386412Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:20.3386907Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:20.3387425Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:20.3387972Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:20.3388584Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:20.3389190Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:20.3389721Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:20.3390302Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:20.3390916Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:20.3391487Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:20.3392002Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:20.3392546Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:20.3393147Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:20.3393745Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:20.3394324Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:20.3394827Z distributed/fsdp/test_utils 2023-11-14T04:55:20.3395278Z distributed/fsdp/test_wrap 2023-11-14T04:55:20.3395755Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:20.3396371Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:20.3396991Z distributed/optim/test_named_optimizer 2023-11-14T04:55:20.3397534Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:20.3398056Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:20.3398783Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:20.3399465Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:20.3400139Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:20.3400755Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:20.3401449Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:20.3402127Z distributed/test_c10d_functional_native 2023-11-14T04:55:20.3402666Z distributed/test_c10d_logger 2023-11-14T04:55:20.3403134Z distributed/test_collective_utils 2023-11-14T04:55:20.3403673Z distributed/test_compute_comm_reordering 2023-11-14T04:55:20.3404223Z distributed/test_device_mesh 2023-11-14T04:55:20.3404695Z distributed/test_distributed_spawn 2023-11-14T04:55:20.3405212Z distributed/test_fake_pg 2023-11-14T04:55:20.3405671Z distributed/test_functional_api 2023-11-14T04:55:20.3420977Z distributed/test_inductor_collectives 2023-11-14T04:55:20.3421652Z distributed/test_multi_threaded_pg 2023-11-14T04:55:20.4180961Z Heuristic Profiling identified 133 tests to prioritize (73.08%%) 2023-11-14T04:55:20.4181820Z Probable Relevance tests (133): 2023-11-14T04:55:20.4182483Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:20.4183292Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:20.4184121Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:20.4185039Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:20.4185970Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:20.4186815Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:20.4187924Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:20.4188634Z distributed/_composable/test_checkpoint 2023-11-14T04:55:20.4189201Z distributed/_composable/test_compose 2023-11-14T04:55:20.4189762Z distributed/_composable/test_contract 2023-11-14T04:55:20.4190326Z distributed/_composable/test_replicate 2023-11-14T04:55:20.4190942Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:20.4191661Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:20.4192448Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:20.4193550Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:20.4194862Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:20.4195701Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:20.4196846Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:20.4197705Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:20.4198760Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:20.4200088Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:20.4201023Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:20.4202227Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:20.4203288Z distributed/_tensor/test_api 2023-11-14T04:55:20.4203802Z distributed/_tensor/test_common_rules 2023-11-14T04:55:20.4204550Z distributed/_tensor/test_dtensor 2023-11-14T04:55:20.4205497Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:20.4206485Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:20.4207171Z distributed/_tensor/test_init 2023-11-14T04:55:20.4208094Z distributed/_tensor/test_math_ops 2023-11-14T04:55:20.4208982Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:20.4209993Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:20.4211433Z distributed/_tensor/test_random_ops 2023-11-14T04:55:20.4212013Z distributed/_tensor/test_redistribute 2023-11-14T04:55:20.4212809Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:20.4213442Z distributed/_tensor/test_utils 2023-11-14T04:55:20.4214086Z distributed/_tensor/test_view_ops 2023-11-14T04:55:20.4215232Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:20.4215878Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:20.4216616Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:20.4217245Z distributed/algorithms/test_join 2023-11-14T04:55:20.4217786Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:20.4218368Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:20.4218985Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:20.4219659Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:20.4220388Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:20.4221065Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:20.4221696Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:20.4222295Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:20.4222900Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:20.4223472Z distributed/checkpoint/test_planner 2023-11-14T04:55:20.4224024Z distributed/checkpoint/test_traverse 2023-11-14T04:55:20.4224547Z distributed/checkpoint/test_utils 2023-11-14T04:55:20.4225074Z distributed/elastic/events/lib_test 2023-11-14T04:55:20.4225611Z distributed/elastic/metrics/api_test 2023-11-14T04:55:20.4226187Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:20.4226826Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:20.4227451Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:20.4228048Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:20.4228640Z distributed/elastic/utils/logging_test 2023-11-14T04:55:20.4229194Z distributed/elastic/utils/util_test 2023-11-14T04:55:20.4229732Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:20.4230478Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:20.4231093Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:20.4231637Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:20.4232234Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:20.4232804Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:20.4233354Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:20.4233869Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:20.4234404Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:20.4234940Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:20.4235536Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:20.4236084Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:20.4236628Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:20.4237252Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:20.4237826Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:20.4238313Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:20.4238874Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:20.4239466Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:20.4240025Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:20.4240551Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:20.4241071Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:20.4241556Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:20.4242101Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:20.4242722Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:20.4243331Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:20.4243937Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:20.4244492Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:20.4245012Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:20.4245605Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:20.4246208Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:20.4246783Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:20.4247353Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:20.4247898Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:20.4248449Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:20.4249168Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:20.4249745Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:20.4250255Z distributed/fsdp/test_utils 2023-11-14T04:55:20.4250951Z distributed/fsdp/test_wrap 2023-11-14T04:55:20.4251459Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:20.4252067Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:20.4252702Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:20.4253277Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:20.4253801Z distributed/rpc/test_share_memory 2023-11-14T04:55:20.4254380Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:20.4254981Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:20.4255677Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:20.4256367Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:20.4257013Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:20.4257669Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:20.4258310Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:20.4258989Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:20.4259647Z distributed/test_c10d_common 2023-11-14T04:55:20.4260119Z distributed/test_c10d_gloo 2023-11-14T04:55:20.4260565Z distributed/test_c10d_logger 2023-11-14T04:55:20.4261034Z distributed/test_c10d_nccl 2023-11-14T04:55:20.4261530Z distributed/test_c10d_object_collectives 2023-11-14T04:55:20.4262060Z distributed/test_c10d_pypg 2023-11-14T04:55:20.4262530Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:20.4263036Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:20.4263522Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:20.4264011Z distributed/test_c10d_ucc 2023-11-14T04:55:20.4264596Z distributed/test_data_parallel 2023-11-14T04:55:20.4265107Z distributed/test_distributed_spawn 2023-11-14T04:55:20.4265645Z distributed/test_dynamo_distributed 2023-11-14T04:55:20.4266169Z distributed/test_fake_pg 2023-11-14T04:55:20.4266611Z distributed/test_functional_api 2023-11-14T04:55:20.4267129Z distributed/test_inductor_collectives 2023-11-14T04:55:20.4267656Z distributed/test_launcher 2023-11-14T04:55:20.4268115Z distributed/test_multi_threaded_pg 2023-11-14T04:55:20.4268624Z distributed/test_nccl 2023-11-14T04:55:20.4269064Z distributed/test_pg_wrapper 2023-11-14T04:55:20.4269509Z distributed/test_store 2023-11-14T04:55:20.4269962Z Unranked Relevance tests (49): 2023-11-14T04:55:20.4270449Z distributed/_shard/test_sharder 2023-11-14T04:55:20.4270947Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:20.4271484Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:20.4272008Z distributed/_spmd/test_tracing 2023-11-14T04:55:20.4272508Z distributed/_spmd/test_transformation 2023-11-14T04:55:20.4273140Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:20.4273770Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:20.4274369Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:20.4274912Z distributed/_tensor/test_optimizers 2023-11-14T04:55:20.4275462Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:20.4276071Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:20.4276724Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:20.4277326Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:20.4277928Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:20.4278653Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:20.4279361Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:20.4279945Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:20.4280538Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:20.4281153Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:20.4281713Z distributed/elastic/timer/api_test 2023-11-14T04:55:20.4282291Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:20.4283119Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:20.4283739Z distributed/optim/test_named_optimizer 2023-11-14T04:55:20.4284305Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:20.4284888Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:20.4285553Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:20.4286230Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:20.4286833Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:20.4287446Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:20.4288080Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:20.4288748Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:20.4289410Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:20.4289954Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:20.4290512Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:20.4291297Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:20.4291904Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:20.4292555Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:20.4293138Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:20.4293699Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:20.4294272Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:20.4294820Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:20.4295358Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:20.4295924Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:20.4296514Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:20.4297087Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:20.4297645Z distributed/test_c10d_functional_native 2023-11-14T04:55:20.4298308Z distributed/test_collective_utils 2023-11-14T04:55:20.4298859Z distributed/test_compute_comm_reordering 2023-11-14T04:55:20.4299417Z distributed/test_device_mesh 2023-11-14T04:55:20.4327416Z High Relevance tests (1): 2023-11-14T04:55:20.4328017Z distributed/test_distributed_spawn 2023-11-14T04:55:20.4328808Z Probable Relevance tests (48): 2023-11-14T04:55:20.4329754Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:20.4330439Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:20.4331430Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:20.4331992Z distributed/test_multi_threaded_pg 2023-11-14T04:55:20.4332487Z distributed/test_c10d_nccl 2023-11-14T04:55:20.4333066Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:20.4333707Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:20.4334250Z distributed/_tensor/test_dtensor 2023-11-14T04:55:20.4334730Z distributed/test_c10d_ucc 2023-11-14T04:55:20.4335193Z distributed/test_pg_wrapper 2023-11-14T04:55:20.4335697Z distributed/test_dynamo_distributed 2023-11-14T04:55:20.4336255Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:20.4336864Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:20.4337423Z distributed/test_c10d_gloo 2023-11-14T04:55:20.4337863Z distributed/test_launcher 2023-11-14T04:55:20.4338573Z distributed/test_inductor_collectives 2023-11-14T04:55:20.4339245Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:20.4339861Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:20.4340435Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:20.4340998Z distributed/test_c10d_common 2023-11-14T04:55:20.4341601Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:20.4342779Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:20.4343628Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:20.4344286Z distributed/test_c10d_object_collectives 2023-11-14T04:55:20.4345302Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:20.4346224Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:20.4347283Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:20.4348243Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:20.4349211Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:20.4350285Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:20.4350940Z distributed/test_data_parallel 2023-11-14T04:55:20.4351441Z distributed/_spmd/test_tracing 2023-11-14T04:55:20.4351963Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:20.4352731Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:20.4353527Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:20.4354097Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:20.4354597Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:20.4355139Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:20.4355712Z distributed/test_compute_comm_reordering 2023-11-14T04:55:20.4356312Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:20.4356922Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:20.4357528Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:20.4358170Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:20.4358714Z distributed/_tensor/test_utils 2023-11-14T04:55:20.4359294Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:20.4360002Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:20.4360655Z distributed/checkpoint/test_planner 2023-11-14T04:55:20.4361303Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:20.4361937Z Unranked Relevance tests (133): 2023-11-14T04:55:20.4362631Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:20.4363558Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:20.4364557Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:20.4365266Z distributed/_composable/test_checkpoint 2023-11-14T04:55:20.4365838Z distributed/_composable/test_compose 2023-11-14T04:55:20.4366394Z distributed/_composable/test_contract 2023-11-14T04:55:20.4366938Z distributed/_composable/test_replicate 2023-11-14T04:55:20.4367585Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:20.4368297Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:20.4369021Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:20.4369728Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:20.4370408Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:20.4371287Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:20.4371965Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:20.4372729Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:20.4373485Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:20.4374191Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:20.4374812Z distributed/_shard/test_sharder 2023-11-14T04:55:20.4375321Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:20.4375859Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:20.4376397Z distributed/_spmd/test_transformation 2023-11-14T04:55:20.4377003Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:20.4377626Z distributed/_tensor/test_api 2023-11-14T04:55:20.4378113Z distributed/_tensor/test_common_rules 2023-11-14T04:55:20.4378648Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:20.4379189Z distributed/_tensor/test_init 2023-11-14T04:55:20.4379685Z distributed/_tensor/test_math_ops 2023-11-14T04:55:20.4380190Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:20.4380728Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:20.4381270Z distributed/_tensor/test_optimizers 2023-11-14T04:55:20.4381798Z distributed/_tensor/test_random_ops 2023-11-14T04:55:20.4382336Z distributed/_tensor/test_redistribute 2023-11-14T04:55:20.4382880Z distributed/_tensor/test_view_ops 2023-11-14T04:55:20.4383559Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:20.4384141Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:20.4384779Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:20.4385492Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:20.4386137Z distributed/algorithms/test_join 2023-11-14T04:55:20.4386713Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:20.4387350Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:20.4387958Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:20.4388540Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:20.4389097Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:20.4389726Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:20.4390379Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:20.4391021Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:20.4391638Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:20.4392355Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:20.4393026Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:20.4393586Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:20.4394179Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:20.4394776Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:20.4395369Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:20.4395947Z distributed/checkpoint/test_traverse 2023-11-14T04:55:20.4396483Z distributed/checkpoint/test_utils 2023-11-14T04:55:20.4396994Z distributed/elastic/events/lib_test 2023-11-14T04:55:20.4397524Z distributed/elastic/metrics/api_test 2023-11-14T04:55:20.4398110Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:20.4398792Z distributed/elastic/timer/api_test 2023-11-14T04:55:20.4399394Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:20.4400018Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:20.4400599Z distributed/elastic/utils/logging_test 2023-11-14T04:55:20.4401146Z distributed/elastic/utils/util_test 2023-11-14T04:55:20.4401724Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:20.4402292Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:20.4402847Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:20.4403436Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:20.4403989Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:20.4404553Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:20.4405071Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:20.4405642Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:20.4406233Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:20.4406814Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:20.4407395Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:20.4407940Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:20.4408457Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:20.4408938Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:20.4409479Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:20.4410099Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:20.4410810Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:20.4411383Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:20.4411952Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:20.4412512Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:20.4413043Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:20.4413565Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:20.4414055Z distributed/fsdp/test_utils 2023-11-14T04:55:20.4414524Z distributed/fsdp/test_wrap 2023-11-14T04:55:20.4415004Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:20.4415664Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:20.4416288Z distributed/optim/test_named_optimizer 2023-11-14T04:55:20.4417052Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:20.4417688Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:20.4418253Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:20.4418934Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:20.4419615Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:20.4420202Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:20.4420829Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:20.4421461Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:20.4422122Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:20.4422832Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:20.4423392Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:20.4423940Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:20.4424514Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:20.4425136Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:20.4425771Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:20.4426353Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:20.4426933Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:20.4427498Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:20.4428044Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:20.4428601Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:20.4429153Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:20.4429739Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:20.4430329Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:20.4430853Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:20.4431370Z distributed/rpc/test_share_memory 2023-11-14T04:55:20.4431953Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:20.4432722Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:20.4433410Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:20.4434071Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:20.4434693Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:20.4435389Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:20.4436079Z distributed/test_c10d_functional_native 2023-11-14T04:55:20.4436618Z distributed/test_c10d_logger 2023-11-14T04:55:20.4437074Z distributed/test_c10d_pypg 2023-11-14T04:55:20.4437549Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:20.4438052Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:20.4438538Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:20.4439042Z distributed/test_collective_utils 2023-11-14T04:55:20.4439553Z distributed/test_device_mesh 2023-11-14T04:55:20.4440006Z distributed/test_fake_pg 2023-11-14T04:55:20.4440465Z distributed/test_functional_api 2023-11-14T04:55:20.4440956Z distributed/test_nccl 2023-11-14T04:55:20.4441365Z distributed/test_store 2023-11-14T04:55:20.4447394Z Found test times from artifacts 2023-11-14T04:55:20.4683562Z Found test times from artifacts 2023-11-14T04:55:20.4738527Z Name: high_relevance 2023-11-14T04:55:20.4739397Z Parallel tests: 2023-11-14T04:55:20.4740159Z Serial tests: 2023-11-14T04:55:20.4741010Z distributed/test_distributed_spawn 1/2 2023-11-14T04:55:20.4742084Z Name: probable_relevance 2023-11-14T04:55:20.4742886Z Parallel tests: 2023-11-14T04:55:20.4743604Z Serial tests: 2023-11-14T04:55:20.4744355Z distributed/fsdp/test_fsdp_optim_state 1/1 2023-11-14T04:55:20.4745521Z distributed/rpc/test_tensorpipe_agent 1/1 2023-11-14T04:55:20.4746646Z distributed/test_multi_threaded_pg 1/1 2023-11-14T04:55:20.4747694Z distributed/test_c10d_nccl 1/1 2023-11-14T04:55:20.4748871Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 2023-11-14T04:55:20.4750209Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2023-11-14T04:55:20.4751502Z distributed/test_c10d_object_collectives 1/1 2023-11-14T04:55:20.4752708Z distributed/fsdp/test_fsdp_grad_acc 1/1 2023-11-14T04:55:20.4754506Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 2023-11-14T04:55:20.4756003Z distributed/_tensor/test_tensor_ops 1/1 2023-11-14T04:55:20.4757205Z distributed/fsdp/test_fsdp_flatten_params 1/1 2023-11-14T04:55:20.4758467Z distributed/fsdp/test_fsdp_exec_order 1/1 2023-11-14T04:55:20.4759711Z distributed/fsdp/test_checkpoint_wrapper 1/1 2023-11-14T04:55:20.4760915Z distributed/test_data_parallel 1/1 2023-11-14T04:55:20.4762054Z distributed/_spmd/test_tracing 1/1 2023-11-14T04:55:20.4763247Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2023-11-14T04:55:20.4764385Z Name: unranked_relevance 2023-11-14T04:55:20.4765244Z Parallel tests: 2023-11-14T04:55:20.4765974Z Serial tests: 2023-11-14T04:55:20.4766865Z distributed/optim/test_zero_redundancy_optimizer 1/1 2023-11-14T04:55:20.4768127Z distributed/fsdp/test_fsdp_checkpoint 1/1 2023-11-14T04:55:20.4769280Z distributed/test_store 1/1 2023-11-14T04:55:20.4770321Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 2023-11-14T04:55:20.4772093Z distributed/fsdp/test_fsdp_comm 1/1 2023-11-14T04:55:20.4772702Z distributed/fsdp/test_fsdp_overlap 1/1 2023-11-14T04:55:20.4773307Z distributed/checkpoint/test_checkpoint 1/1 2023-11-14T04:55:20.4773915Z distributed/_tensor/test_matrix_ops 1/1 2023-11-14T04:55:20.4774485Z distributed/_tensor/test_init 1/1 2023-11-14T04:55:20.4775039Z distributed/test_c10d_functional_native 1/1 2023-11-14T04:55:20.4775706Z distributed/checkpoint/test_dtensor_resharding 1/1 2023-11-14T04:55:20.4776420Z distributed/tensor/parallel/test_tp_examples 1/1 2023-11-14T04:55:20.4777084Z distributed/fsdp/test_fsdp_tp_integration 1/1 2023-11-14T04:55:20.4777691Z distributed/test_c10d_spawn_ucc 1/1 2023-11-14T04:55:20.4778248Z distributed/fsdp/test_fsdp_apply 1/1 2023-11-14T04:55:20.4779083Z distributed/checkpoint/test_state_dict_utils 1/1 2023-11-14T04:55:20.4779820Z distributed/checkpoint/test_fsdp_optim_state 1/1 2023-11-14T04:55:20.4780530Z distributed/checkpoint/test_fsdp_model_state 1/1 2023-11-14T04:55:20.4781213Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2023-11-14T04:55:20.4781914Z distributed/elastic/timer/local_timer_example 1/1 2023-11-14T04:55:20.4782703Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2023-11-14T04:55:20.4783571Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2023-11-14T04:55:20.4784338Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2023-11-14T04:55:20.4785005Z distributed/fsdp/test_fsdp_traversal 1/1 2023-11-14T04:55:20.4785657Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2023-11-14T04:55:20.4786310Z distributed/elastic/timer/local_timer_test 1/1 2023-11-14T04:55:20.4786928Z distributed/rpc/test_share_memory 1/1 2023-11-14T04:55:20.4787526Z distributed/pipeline/sync/test_stream 1/1 2023-11-14T04:55:20.4788114Z distributed/pipeline/sync/test_copy 1/1 2023-11-14T04:55:20.4788728Z distributed/pipeline/sync/skip/test_leak 1/1 2023-11-14T04:55:20.4789361Z distributed/_tensor/test_op_strategy 1/1 2023-11-14T04:55:20.4789935Z distributed/elastic/events/lib_test 1/1 2023-11-14T04:55:20.4790532Z distributed/nn/jit/test_instantiator 1/1 2023-11-14T04:55:20.4791181Z distributed/_shard/sharded_tensor/test_logger 1/1 2023-11-14T04:55:20.4791813Z distributed/checkpoint/test_traverse 1/1 2023-11-14T04:55:20.4792429Z distributed/_composable/test_checkpoint 1/1 2023-11-14T04:55:20.4793074Z distributed/pipeline/sync/skip/test_portal 1/1 2023-11-14T04:55:20.4793697Z distributed/pipeline/sync/test_phony 1/1 2023-11-14T04:55:20.4794285Z distributed/_spmd/test_graph_utils 1/1 2023-11-14T04:55:20.4794870Z distributed/_tensor/test_optimizers 1/1 2023-11-14T04:55:20.4795536Z distributed/optim/test_apply_optimizer_in_backward 1/1 2023-11-14T04:55:20.4796205Z distributed/test_collective_utils 1/1 2023-11-14T04:55:20.4796739Z Name: unlikely_relevance 2023-11-14T04:55:20.4797150Z Parallel tests: 2023-11-14T04:55:20.4797727Z Serial tests: 2023-11-14T04:55:20.4798104Z Name: none_relevance 2023-11-14T04:55:20.4798487Z Parallel tests: 2023-11-14T04:55:20.4798862Z Serial tests: 2023-11-14T04:55:20.4799784Z Starting test batch 'high_relevance' 1.9073486328125e-06 seconds after initiating testing 2023-11-14T04:55:20.4800652Z With sharding, this batch will run 1 tests 2023-11-14T04:55:22.5291570Z Running distributed/test_distributed_spawn 1/2 ... [2023-11-14 04:55:22.528536] 2023-11-14T04:55:22.5299160Z /usr/bin/mpiexec 2023-11-14T04:55:22.5300011Z MPI not available -- MPI backend tests will be skipped 2023-11-14T04:55:22.5305162Z Running distributed tests for the test backend with env init_method 2023-11-14T04:55:22.5311754Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=0', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 04:55:22.530728] 2023-11-14T04:55:27.9795621Z 2023-11-14T04:55:27.9798129Z distributed/test_distributed_spawn 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.2_6954fa4c4f370602_.log 2023-11-14T04:55:27.9799523Z 2023-11-14T04:55:27.9809687Z Running distributed tests for the test backend with file init_method 2023-11-14T04:55:27.9817206Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=0', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 04:55:27.981190] 2023-11-14T04:55:33.2503402Z 2023-11-14T04:55:33.2506363Z distributed/test_distributed_spawn 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.2_d243e10469c8e9c2_.log 2023-11-14T04:55:33.2507758Z 2023-11-14T04:55:33.2514689Z Running distributed tests for the nccl backend with env init_method 2023-11-14T04:55:33.2521230Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=0', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 04:55:33.251597] 2023-11-14T05:16:37.9717082Z 2023-11-14T05:16:37.9719701Z distributed/test_distributed_spawn 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.2_a419aa61a4bf8873_.log 2023-11-14T05:16:37.9722564Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2023-11-14T05:16:37.9726091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2023-11-14T05:16:37.9728480Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2023-11-14T05:16:37.9730504Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2023-11-14T05:16:37.9732872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2023-11-14T05:16:37.9736596Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2023-11-14T05:16:37.9738637Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2023-11-14T05:16:37.9740665Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2023-11-14T05:16:37.9745377Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2023-11-14T05:16:37.9748605Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2023-11-14T05:16:37.9750483Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2023-11-14T05:16:37.9752268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2023-11-14T05:16:37.9754052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2023-11-14T05:16:37.9755751Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2023-11-14T05:16:37.9757476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2023-11-14T05:16:37.9759289Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2023-11-14T05:16:37.9761093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu 2023-11-14T05:16:37.9762881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2023-11-14T05:16:37.9764702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2023-11-14T05:16:37.9766753Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2023-11-14T05:16:37.9768606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2023-11-14T05:16:37.9770421Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2023-11-14T05:16:37.9772554Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2023-11-14T05:16:37.9774350Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2023-11-14T05:16:37.9776178Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2023-11-14T05:16:37.9778092Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2023-11-14T05:16:37.9779896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2023-11-14T05:16:37.9781626Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2023-11-14T05:16:37.9783360Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2023-11-14T05:16:37.9785068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2023-11-14T05:16:37.9786776Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2023-11-14T05:16:37.9788426Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2023-11-14T05:16:37.9790066Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2023-11-14T05:16:37.9791912Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2023-11-14T05:16:37.9793631Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2023-11-14T05:16:37.9795354Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2023-11-14T05:16:37.9797053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2023-11-14T05:16:37.9798677Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2023-11-14T05:16:37.9800338Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2023-11-14T05:16:37.9802049Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2023-11-14T05:16:37.9803771Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2023-11-14T05:16:37.9805498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2023-11-14T05:16:37.9807324Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2023-11-14T05:16:37.9809168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2023-11-14T05:16:37.9811497Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2023-11-14T05:16:37.9813485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2023-11-14T05:16:37.9815407Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2023-11-14T05:16:37.9817268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2023-11-14T05:16:37.9819149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2023-11-14T05:16:37.9821057Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2023-11-14T05:16:37.9823005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2023-11-14T05:16:37.9824779Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2023-11-14T05:16:37.9826409Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2023-11-14T05:16:37.9827991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2023-11-14T05:16:37.9829586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2023-11-14T05:16:37.9831293Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2023-11-14T05:16:37.9833019Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2023-11-14T05:16:37.9834732Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2023-11-14T05:16:37.9836606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2023-11-14T05:16:37.9838433Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2023-11-14T05:16:37.9840132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2023-11-14T05:16:37.9841740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2023-11-14T05:16:37.9843378Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2023-11-14T05:16:37.9845015Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_multigpu 2023-11-14T05:16:37.9846692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2023-11-14T05:16:37.9848368Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2023-11-14T05:16:37.9850040Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2023-11-14T05:16:37.9852249Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2023-11-14T05:16:37.9854360Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2023-11-14T05:16:37.9856477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2023-11-14T05:16:37.9858295Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2023-11-14T05:16:37.9860087Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2023-11-14T05:16:37.9861860Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2023-11-14T05:16:37.9863750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2023-11-14T05:16:37.9865663Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2023-11-14T05:16:37.9867448Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2023-11-14T05:16:37.9869283Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2023-11-14T05:16:37.9871139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2023-11-14T05:16:37.9872806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2023-11-14T05:16:37.9874472Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2023-11-14T05:16:37.9876185Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2023-11-14T05:16:37.9877848Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2023-11-14T05:16:37.9879616Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2023-11-14T05:16:37.9881715Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2023-11-14T05:16:37.9883834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2023-11-14T05:16:37.9886016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2023-11-14T05:16:37.9888233Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2023-11-14T05:16:37.9890180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2023-11-14T05:16:37.9892214Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2023-11-14T05:16:37.9894013Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2023-11-14T05:16:37.9895822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2023-11-14T05:16:37.9897578Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2023-11-14T05:16:37.9899272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2023-11-14T05:16:37.9901169Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2023-11-14T05:16:37.9903035Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2023-11-14T05:16:37.9904734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2023-11-14T05:16:37.9906453Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2023-11-14T05:16:37.9908215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2023-11-14T05:16:37.9909978Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2023-11-14T05:16:37.9911648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2023-11-14T05:16:37.9913231Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2023-11-14T05:16:37.9914798Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2023-11-14T05:16:37.9916367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2023-11-14T05:16:37.9917990Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2023-11-14T05:16:37.9919667Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2023-11-14T05:16:37.9921397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2023-11-14T05:16:37.9923248Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2023-11-14T05:16:37.9925093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2023-11-14T05:16:37.9926993Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2023-11-14T05:16:37.9928750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2023-11-14T05:16:37.9930414Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2023-11-14T05:16:37.9932318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2023-11-14T05:16:37.9934199Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2023-11-14T05:16:37.9936195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2023-11-14T05:16:37.9938099Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2023-11-14T05:16:37.9939955Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2023-11-14T05:16:37.9941844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2023-11-14T05:16:37.9943666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2023-11-14T05:16:37.9945490Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2023-11-14T05:16:37.9947225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2023-11-14T05:16:37.9948830Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2023-11-14T05:16:37.9950481Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2023-11-14T05:16:37.9952154Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2023-11-14T05:16:37.9953764Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2023-11-14T05:16:37.9955307Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2023-11-14T05:16:37.9956898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2023-11-14T05:16:37.9958500Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2023-11-14T05:16:37.9960272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2023-11-14T05:16:37.9962086Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2023-11-14T05:16:37.9963760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2023-11-14T05:16:37.9965417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2023-11-14T05:16:37.9967080Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2023-11-14T05:16:37.9968753Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2023-11-14T05:16:37.9970621Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2023-11-14T05:16:37.9972560Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2023-11-14T05:16:37.9974293Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2023-11-14T05:16:37.9976181Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2023-11-14T05:16:37.9978071Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2023-11-14T05:16:37.9979123Z 2023-11-14T05:16:37.9979513Z Running distributed tests for the nccl backend with file init_method 2023-11-14T05:16:37.9981975Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=0', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 05:16:37.976280] 2023-11-14T05:37:42.0392538Z 2023-11-14T05:37:42.0395319Z distributed/test_distributed_spawn 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.2_02b673c50f1d3d59_.log 2023-11-14T05:37:42.0397660Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2023-11-14T05:37:42.0399789Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2023-11-14T05:37:42.0403912Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2023-11-14T05:37:42.0406185Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2023-11-14T05:37:42.0408751Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2023-11-14T05:37:42.0411208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2023-11-14T05:37:42.0414666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2023-11-14T05:37:42.0418311Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2023-11-14T05:37:42.0420734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2023-11-14T05:37:42.0422498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2023-11-14T05:37:42.0424181Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2023-11-14T05:37:42.0425996Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2023-11-14T05:37:42.0428358Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2023-11-14T05:37:42.0432031Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2023-11-14T05:37:42.0433800Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2023-11-14T05:37:42.0435645Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2023-11-14T05:37:42.0437681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu 2023-11-14T05:37:42.0439410Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2023-11-14T05:37:42.0441240Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2023-11-14T05:37:42.0443106Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2023-11-14T05:37:42.0444962Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2023-11-14T05:37:42.0446760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2023-11-14T05:37:42.0448600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2023-11-14T05:37:42.0450401Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2023-11-14T05:37:42.0452831Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2023-11-14T05:37:42.0454738Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2023-11-14T05:37:42.0456702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2023-11-14T05:37:42.0458475Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2023-11-14T05:37:42.0460231Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2023-11-14T05:37:42.0461927Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2023-11-14T05:37:42.0463642Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2023-11-14T05:37:42.0465309Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2023-11-14T05:37:42.0466966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2023-11-14T05:37:42.0468650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2023-11-14T05:37:42.0470392Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2023-11-14T05:37:42.0472148Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2023-11-14T05:37:42.0473837Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2023-11-14T05:37:42.0475465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2023-11-14T05:37:42.0477136Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2023-11-14T05:37:42.0478880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2023-11-14T05:37:42.0480676Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2023-11-14T05:37:42.0482560Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2023-11-14T05:37:42.0484405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2023-11-14T05:37:42.0486270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2023-11-14T05:37:42.0488146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2023-11-14T05:37:42.0490117Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2023-11-14T05:37:42.0492649Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2023-11-14T05:37:42.0494570Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2023-11-14T05:37:42.0496523Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2023-11-14T05:37:42.0498428Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2023-11-14T05:37:42.0500387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2023-11-14T05:37:42.0502334Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2023-11-14T05:37:42.0503994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2023-11-14T05:37:42.0505576Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2023-11-14T05:37:42.0507193Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2023-11-14T05:37:42.0508906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2023-11-14T05:37:42.0510644Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2023-11-14T05:37:42.0512349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2023-11-14T05:37:42.0514120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2023-11-14T05:37:42.0515962Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2023-11-14T05:37:42.0517671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2023-11-14T05:37:42.0519266Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2023-11-14T05:37:42.0520918Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2023-11-14T05:37:42.0522557Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_multigpu 2023-11-14T05:37:42.0524226Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2023-11-14T05:37:42.0525913Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2023-11-14T05:37:42.0527769Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2023-11-14T05:37:42.0529716Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2023-11-14T05:37:42.0532388Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2023-11-14T05:37:42.0534417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2023-11-14T05:37:42.0536246Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2023-11-14T05:37:42.0538004Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2023-11-14T05:37:42.0539774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2023-11-14T05:37:42.0541686Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2023-11-14T05:37:42.0543630Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2023-11-14T05:37:42.0545417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2023-11-14T05:37:42.0547383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2023-11-14T05:37:42.0549263Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2023-11-14T05:37:42.0550964Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2023-11-14T05:37:42.0552637Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2023-11-14T05:37:42.0554341Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2023-11-14T05:37:42.0556025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2023-11-14T05:37:42.0557801Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2023-11-14T05:37:42.0559756Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2023-11-14T05:37:42.0561874Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2023-11-14T05:37:42.0564088Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2023-11-14T05:37:42.0566314Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2023-11-14T05:37:42.0568274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2023-11-14T05:37:42.0570008Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2023-11-14T05:37:42.0572434Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2023-11-14T05:37:42.0574402Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2023-11-14T05:37:42.0576176Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2023-11-14T05:37:42.0577868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2023-11-14T05:37:42.0579676Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2023-11-14T05:37:42.0581540Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2023-11-14T05:37:42.0583247Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2023-11-14T05:37:42.0584965Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2023-11-14T05:37:42.0586759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2023-11-14T05:37:42.0588532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2023-11-14T05:37:42.0590211Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2023-11-14T05:37:42.0591790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2023-11-14T05:37:42.0593481Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2023-11-14T05:37:42.0595085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2023-11-14T05:37:42.0596756Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2023-11-14T05:37:42.0598444Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2023-11-14T05:37:42.0600210Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2023-11-14T05:37:42.0602071Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2023-11-14T05:37:42.0603920Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2023-11-14T05:37:42.0605726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2023-11-14T05:37:42.0607510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2023-11-14T05:37:42.0609189Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2023-11-14T05:37:42.0611417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2023-11-14T05:37:42.0613726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2023-11-14T05:37:42.0615738Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2023-11-14T05:37:42.0617655Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2023-11-14T05:37:42.0619672Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2023-11-14T05:37:42.0621588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2023-11-14T05:37:42.0623418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2023-11-14T05:37:42.0625144Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2023-11-14T05:37:42.0626844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2023-11-14T05:37:42.0628465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2023-11-14T05:37:42.0630127Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2023-11-14T05:37:42.0631818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2023-11-14T05:37:42.0633421Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2023-11-14T05:37:42.0634994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2023-11-14T05:37:42.0636590Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2023-11-14T05:37:42.0638295Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2023-11-14T05:37:42.0640034Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2023-11-14T05:37:42.0641884Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2023-11-14T05:37:42.0643578Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2023-11-14T05:37:42.0645232Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2023-11-14T05:37:42.0646926Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2023-11-14T05:37:42.0648597Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2023-11-14T05:37:42.0650333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2023-11-14T05:37:42.0652291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2023-11-14T05:37:42.0654060Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2023-11-14T05:37:42.0655961Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2023-11-14T05:37:42.0657858Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2023-11-14T05:37:42.0658913Z 2023-11-14T05:37:42.0659271Z Running distributed tests for the gloo backend with env init_method 2023-11-14T05:37:42.0661755Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=0', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 05:37:42.045077] 2023-11-14T06:00:04.6445709Z 2023-11-14T06:00:04.6448321Z distributed/test_distributed_spawn 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.2_ce38aa1f69712c59_.log 2023-11-14T06:00:04.6451938Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2023-11-14T06:00:04.6455512Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2023-11-14T06:00:04.6457586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2023-11-14T06:00:04.6460906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2023-11-14T06:00:04.6463359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2023-11-14T06:00:04.6465514Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2023-11-14T06:00:04.6469836Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2023-11-14T06:00:04.6473643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2023-11-14T06:00:04.6477022Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2023-11-14T06:00:04.6479085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2023-11-14T06:00:04.6480780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2023-11-14T06:00:04.6482567Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2023-11-14T06:00:04.6484359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2023-11-14T06:00:04.6486043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2023-11-14T06:00:04.6487767Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2023-11-14T06:00:04.6489664Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2023-11-14T06:00:04.6491765Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu 2023-11-14T06:00:04.6493517Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2023-11-14T06:00:04.6495343Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2023-11-14T06:00:04.6497205Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2023-11-14T06:00:04.6499028Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2023-11-14T06:00:04.6500851Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2023-11-14T06:00:04.6502695Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2023-11-14T06:00:04.6504744Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2023-11-14T06:00:04.6506588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2023-11-14T06:00:04.6508557Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2023-11-14T06:00:04.6510368Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2023-11-14T06:00:04.6512120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2023-11-14T06:00:04.6513837Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2023-11-14T06:00:04.6515561Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2023-11-14T06:00:04.6517272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2023-11-14T06:00:04.6518934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2023-11-14T06:00:04.6520560Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2023-11-14T06:00:04.6522382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2023-11-14T06:00:04.6524133Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2023-11-14T06:00:04.6525892Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2023-11-14T06:00:04.6527581Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2023-11-14T06:00:04.6529213Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2023-11-14T06:00:04.6531201Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2023-11-14T06:00:04.6532936Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2023-11-14T06:00:04.6534658Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2023-11-14T06:00:04.6536394Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2023-11-14T06:00:04.6538240Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2023-11-14T06:00:04.6540128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2023-11-14T06:00:04.6542026Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2023-11-14T06:00:04.6543982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2023-11-14T06:00:04.6545904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2023-11-14T06:00:04.6547780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2023-11-14T06:00:04.6549827Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2023-11-14T06:00:04.6551748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2023-11-14T06:00:04.6553699Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2023-11-14T06:00:04.6555484Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2023-11-14T06:00:04.6557115Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2023-11-14T06:00:04.6558702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2023-11-14T06:00:04.6560325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2023-11-14T06:00:04.6562016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2023-11-14T06:00:04.6563742Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2023-11-14T06:00:04.6565463Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2023-11-14T06:00:04.6567333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2023-11-14T06:00:04.6569171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2023-11-14T06:00:04.6571214Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2023-11-14T06:00:04.6572836Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2023-11-14T06:00:04.6574481Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2023-11-14T06:00:04.6576126Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_multigpu 2023-11-14T06:00:04.6577785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2023-11-14T06:00:04.6579467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2023-11-14T06:00:04.6581164Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2023-11-14T06:00:04.6583120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2023-11-14T06:00:04.6585215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2023-11-14T06:00:04.6587229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2023-11-14T06:00:04.6589053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2023-11-14T06:00:04.6590802Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2023-11-14T06:00:04.6592564Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2023-11-14T06:00:04.6594604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2023-11-14T06:00:04.6596544Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2023-11-14T06:00:04.6598307Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2023-11-14T06:00:04.6600141Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2023-11-14T06:00:04.6602001Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2023-11-14T06:00:04.6603694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2023-11-14T06:00:04.6605358Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2023-11-14T06:00:04.6607086Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2023-11-14T06:00:04.6608779Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2023-11-14T06:00:04.6610546Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2023-11-14T06:00:04.6613013Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2023-11-14T06:00:04.6615172Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2023-11-14T06:00:04.6617387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2023-11-14T06:00:04.6619608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2023-11-14T06:00:04.6621579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2023-11-14T06:00:04.6623325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2023-11-14T06:00:04.6625142Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2023-11-14T06:00:04.6626953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2023-11-14T06:00:04.6628712Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2023-11-14T06:00:04.6630411Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2023-11-14T06:00:04.6632217Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2023-11-14T06:00:04.6634063Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2023-11-14T06:00:04.6635746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2023-11-14T06:00:04.6637474Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2023-11-14T06:00:04.6639399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2023-11-14T06:00:04.6641219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2023-11-14T06:00:04.6642876Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2023-11-14T06:00:04.6644462Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2023-11-14T06:00:04.6646049Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2023-11-14T06:00:04.6647631Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2023-11-14T06:00:04.6649246Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2023-11-14T06:00:04.6651258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2023-11-14T06:00:04.6653028Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2023-11-14T06:00:04.6654869Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2023-11-14T06:00:04.6656731Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2023-11-14T06:00:04.6658640Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2023-11-14T06:00:04.6660440Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2023-11-14T06:00:04.6662105Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2023-11-14T06:00:04.6663793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2023-11-14T06:00:04.6665695Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2023-11-14T06:00:04.6667682Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2023-11-14T06:00:04.6669584Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2023-11-14T06:00:04.6671448Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2023-11-14T06:00:04.6673371Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2023-11-14T06:00:04.6675202Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2023-11-14T06:00:04.6676903Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2023-11-14T06:00:04.6678620Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2023-11-14T06:00:04.6680242Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2023-11-14T06:00:04.6681898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2023-11-14T06:00:04.6683707Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2023-11-14T06:00:04.6685332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2023-11-14T06:00:04.6686898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2023-11-14T06:00:04.6688494Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2023-11-14T06:00:04.6690087Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2023-11-14T06:00:04.6692115Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2023-11-14T06:00:04.6693960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2023-11-14T06:00:04.6695630Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2023-11-14T06:00:04.6697299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2023-11-14T06:00:04.6698993Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2023-11-14T06:00:04.6700661Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2023-11-14T06:00:04.6702369Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2023-11-14T06:00:04.6704207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2023-11-14T06:00:04.6706000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2023-11-14T06:00:04.6707902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2023-11-14T06:00:04.6709775Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2023-11-14T06:00:04.6710844Z 2023-11-14T06:00:04.6711222Z Running distributed tests for the gloo backend with file init_method 2023-11-14T06:00:04.6713711Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=0', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:00:04.649677] 2023-11-14T06:22:28.3410567Z 2023-11-14T06:22:28.3413847Z distributed/test_distributed_spawn 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.2_49769ddc22e23e17_.log 2023-11-14T06:22:28.3417205Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2023-11-14T06:22:28.3419554Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2023-11-14T06:22:28.3421830Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2023-11-14T06:22:28.3423851Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2023-11-14T06:22:28.3426146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2023-11-14T06:22:28.3428604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2023-11-14T06:22:28.3430619Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2023-11-14T06:22:28.3432636Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2023-11-14T06:22:28.3434925Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2023-11-14T06:22:28.3436703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2023-11-14T06:22:28.3438964Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2023-11-14T06:22:28.3441036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2023-11-14T06:22:28.3442839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2023-11-14T06:22:28.3444572Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2023-11-14T06:22:28.3446291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2023-11-14T06:22:28.3448124Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2023-11-14T06:22:28.3450049Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu 2023-11-14T06:22:28.3452142Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2023-11-14T06:22:28.3453985Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2023-11-14T06:22:28.3455853Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2023-11-14T06:22:28.3457677Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2023-11-14T06:22:28.3459493Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2023-11-14T06:22:28.3461336Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2023-11-14T06:22:28.3463119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2023-11-14T06:22:28.3464983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2023-11-14T06:22:28.3466904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2023-11-14T06:22:28.3468707Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2023-11-14T06:22:28.3470435Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2023-11-14T06:22:28.3475820Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2023-11-14T06:22:28.3479455Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2023-11-14T06:22:28.3483108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2023-11-14T06:22:28.3486569Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2023-11-14T06:22:28.3489910Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2023-11-14T06:22:28.3493796Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2023-11-14T06:22:28.3497458Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2023-11-14T06:22:28.3501097Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2023-11-14T06:22:28.3504610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2023-11-14T06:22:28.3507996Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2023-11-14T06:22:28.3511485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2023-11-14T06:22:28.3515104Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2023-11-14T06:22:28.3518697Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2023-11-14T06:22:28.3522606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2023-11-14T06:22:28.3526426Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2023-11-14T06:22:28.3530278Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2023-11-14T06:22:28.3534701Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2023-11-14T06:22:28.3538811Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2023-11-14T06:22:28.3542726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2023-11-14T06:22:28.3546513Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2023-11-14T06:22:28.3550447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2023-11-14T06:22:28.3554407Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2023-11-14T06:22:28.3558511Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2023-11-14T06:22:28.3562224Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2023-11-14T06:22:28.3565628Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2023-11-14T06:22:28.3568901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2023-11-14T06:22:28.3572715Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2023-11-14T06:22:28.3576505Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2023-11-14T06:22:28.3580130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2023-11-14T06:22:28.3583771Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2023-11-14T06:22:28.3587488Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2023-11-14T06:22:28.3591344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2023-11-14T06:22:28.3594875Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2023-11-14T06:22:28.3598201Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2023-11-14T06:22:28.3601581Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2023-11-14T06:22:28.3605006Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_multigpu 2023-11-14T06:22:28.3608490Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2023-11-14T06:22:28.3612375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2023-11-14T06:22:28.3616198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2023-11-14T06:22:28.3620332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2023-11-14T06:22:28.3624702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2023-11-14T06:22:28.3628907Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2023-11-14T06:22:28.3632665Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2023-11-14T06:22:28.3636330Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2023-11-14T06:22:28.3639939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2023-11-14T06:22:28.3643679Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2023-11-14T06:22:28.3647689Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2023-11-14T06:22:28.3651759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2023-11-14T06:22:28.3655574Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2023-11-14T06:22:28.3659340Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2023-11-14T06:22:28.3662837Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2023-11-14T06:22:28.3666300Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2023-11-14T06:22:28.3670162Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2023-11-14T06:22:28.3673647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2023-11-14T06:22:28.3677255Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2023-11-14T06:22:28.3681387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2023-11-14T06:22:28.3685821Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2023-11-14T06:22:28.3690362Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2023-11-14T06:22:28.3695425Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2023-11-14T06:22:28.3699614Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2023-11-14T06:22:28.3703215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2023-11-14T06:22:28.3706987Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2023-11-14T06:22:28.3711139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2023-11-14T06:22:28.3714815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2023-11-14T06:22:28.3718365Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2023-11-14T06:22:28.3722117Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2023-11-14T06:22:28.3726056Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2023-11-14T06:22:28.3729572Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2023-11-14T06:22:28.3733488Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2023-11-14T06:22:28.3737218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2023-11-14T06:22:28.3740939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2023-11-14T06:22:28.3744420Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2023-11-14T06:22:28.3747703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2023-11-14T06:22:28.3750966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2023-11-14T06:22:28.3754195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2023-11-14T06:22:28.3757578Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2023-11-14T06:22:28.3761113Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2023-11-14T06:22:28.3765017Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2023-11-14T06:22:28.3768886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2023-11-14T06:22:28.3773146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2023-11-14T06:22:28.3776957Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2023-11-14T06:22:28.3780655Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2023-11-14T06:22:28.3784141Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2023-11-14T06:22:28.3787769Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2023-11-14T06:22:28.3791743Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2023-11-14T06:22:28.3795883Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2023-11-14T06:22:28.3799901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2023-11-14T06:22:28.3804058Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2023-11-14T06:22:28.3808064Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2023-11-14T06:22:28.3812208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2023-11-14T06:22:28.3815759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2023-11-14T06:22:28.3819332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2023-11-14T06:22:28.3822760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2023-11-14T06:22:28.3826321Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2023-11-14T06:22:28.3829878Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2023-11-14T06:22:28.3833208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2023-11-14T06:22:28.3836457Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2023-11-14T06:22:28.3839784Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2023-11-14T06:22:28.3843141Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2023-11-14T06:22:28.3846708Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2023-11-14T06:22:28.3850528Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2023-11-14T06:22:28.3854297Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2023-11-14T06:22:28.3858022Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2023-11-14T06:22:28.3861519Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2023-11-14T06:22:28.3864952Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2023-11-14T06:22:28.3868583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2023-11-14T06:22:28.3872124Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2023-11-14T06:22:28.3875891Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2023-11-14T06:22:28.3879939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2023-11-14T06:22:28.3883870Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2023-11-14T06:22:28.3886209Z 2023-11-14T06:22:28.3886986Z Running distributed tests for the ucc backend with env init_method 2023-11-14T06:22:28.3892302Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=0', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:22:28.346845] 2023-11-14T06:44:00.8687031Z 2023-11-14T06:44:00.8689375Z PRINTING LOG FILE of distributed/test_distributed_spawn 1/2 (/var/lib/jenkins/workspace/test/test-reports/distributed.test_distributed_spawn_1.2_o9c4kvs8_toprint.log) 2023-11-14T06:44:00.8691354Z Running 138 items in this shard 2023-11-14T06:44:00.8693299Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2023-11-14T06:44:00.8694808Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2023-11-14T06:44:00.8697158Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2023-11-14T06:44:00.8700422Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2023-11-14T06:44:00.8703327Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2023-11-14T06:44:00.8704998Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2023-11-14T06:44:00.8706645Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2023-11-14T06:44:00.8708343Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2023-11-14T06:44:00.8709976Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2023-11-14T06:44:00.8711943Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2023-11-14T06:44:00.8714735Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2023-11-14T06:44:00.8717656Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2023-11-14T06:44:00.8720288Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2023-11-14T06:44:00.8722753Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2023-11-14T06:44:00.8725342Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2023-11-14T06:44:00.8728503Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2023-11-14T06:44:00.8731521Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu 2023-11-14T06:44:00.8734041Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2023-11-14T06:44:00.8736783Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2023-11-14T06:44:00.8739531Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2023-11-14T06:44:00.8742312Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2023-11-14T06:44:00.8745077Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2023-11-14T06:44:00.8747791Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2023-11-14T06:44:00.8750500Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2023-11-14T06:44:00.8753364Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2023-11-14T06:44:00.8756263Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2023-11-14T06:44:00.8758954Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2023-11-14T06:44:00.8761519Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2023-11-14T06:44:00.8766124Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2023-11-14T06:44:00.8768735Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2023-11-14T06:44:00.8771656Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2023-11-14T06:44:00.8774095Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2023-11-14T06:44:00.8776477Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2023-11-14T06:44:00.8778971Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2023-11-14T06:44:00.8781258Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2023-11-14T06:44:00.8783134Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2023-11-14T06:44:00.8784948Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2023-11-14T06:44:00.8787168Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2023-11-14T06:44:00.8789266Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2023-11-14T06:44:00.8791829Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2023-11-14T06:44:00.8794335Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2023-11-14T06:44:00.8796825Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2023-11-14T06:44:00.8799547Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2023-11-14T06:44:00.8802320Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2023-11-14T06:44:00.8805087Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2023-11-14T06:44:00.8807773Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2023-11-14T06:44:00.8809512Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2023-11-14T06:44:00.8811774Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2023-11-14T06:44:00.8813297Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2023-11-14T06:44:00.8814836Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2023-11-14T06:44:00.8816411Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2023-11-14T06:44:00.8817840Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2023-11-14T06:44:00.8819060Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2023-11-14T06:44:00.8820266Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2023-11-14T06:44:00.8821501Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2023-11-14T06:44:00.8822806Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2023-11-14T06:44:00.8824203Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2023-11-14T06:44:00.8825546Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2023-11-14T06:44:00.8826926Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2023-11-14T06:44:00.8828495Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2023-11-14T06:44:00.8829845Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2023-11-14T06:44:00.8831089Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2023-11-14T06:44:00.8832341Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2023-11-14T06:44:00.8833594Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_multigpu 2023-11-14T06:44:00.8834887Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2023-11-14T06:44:00.8836182Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2023-11-14T06:44:00.8837472Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2023-11-14T06:44:00.8839040Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2023-11-14T06:44:00.8840778Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2023-11-14T06:44:00.8842405Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2023-11-14T06:44:00.8843848Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2023-11-14T06:44:00.8845211Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2023-11-14T06:44:00.8846594Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2023-11-14T06:44:00.8848097Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2023-11-14T06:44:00.8849660Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2023-11-14T06:44:00.8851641Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2023-11-14T06:44:00.8853269Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2023-11-14T06:44:00.8854726Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2023-11-14T06:44:00.8856031Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2023-11-14T06:44:00.8857316Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2023-11-14T06:44:00.8858633Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2023-11-14T06:44:00.8859929Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2023-11-14T06:44:00.8861318Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2023-11-14T06:44:00.8862896Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2023-11-14T06:44:00.8864634Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2023-11-14T06:44:00.8866461Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2023-11-14T06:44:00.8868305Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2023-11-14T06:44:00.8869861Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2023-11-14T06:44:00.8871221Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2023-11-14T06:44:00.8872761Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2023-11-14T06:44:00.8874210Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:00.8875586Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2023-11-14T06:44:00.8876902Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2023-11-14T06:44:00.8878332Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2023-11-14T06:44:00.8879801Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2023-11-14T06:44:00.8881095Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2023-11-14T06:44:00.8882437Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2023-11-14T06:44:00.8883843Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2023-11-14T06:44:00.8885205Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2023-11-14T06:44:00.8886496Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2023-11-14T06:44:00.8887689Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2023-11-14T06:44:00.8888861Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2023-11-14T06:44:00.8890046Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2023-11-14T06:44:00.8891855Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2023-11-14T06:44:00.8893157Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2023-11-14T06:44:00.8894515Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2023-11-14T06:44:00.8895990Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2023-11-14T06:44:00.8897641Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2023-11-14T06:44:00.8899055Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2023-11-14T06:44:00.8900421Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2023-11-14T06:44:00.8901706Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2023-11-14T06:44:00.8903005Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2023-11-14T06:44:00.8904510Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2023-11-14T06:44:00.8906122Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2023-11-14T06:44:00.8907657Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2023-11-14T06:44:00.8909135Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2023-11-14T06:44:00.8910657Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2023-11-14T06:44:00.8912100Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2023-11-14T06:44:00.8913434Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2023-11-14T06:44:00.8914749Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2023-11-14T06:44:00.8916092Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2023-11-14T06:44:00.8917385Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2023-11-14T06:44:00.8918691Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2023-11-14T06:44:00.8919901Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2023-11-14T06:44:00.8921079Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2023-11-14T06:44:00.8922288Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2023-11-14T06:44:00.8923486Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2023-11-14T06:44:00.8924879Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2023-11-14T06:44:00.8926338Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2023-11-14T06:44:00.8927633Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2023-11-14T06:44:00.8928903Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2023-11-14T06:44:00.8930208Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2023-11-14T06:44:00.8932039Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2023-11-14T06:44:00.8933365Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2023-11-14T06:44:00.8934645Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2023-11-14T06:44:00.8936022Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2023-11-14T06:44:00.8937541Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2023-11-14T06:44:00.8939031Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2023-11-14T06:44:00.8940071Z 2023-11-14T06:44:00.8940236Z 284 tests collected in 0.03s 2023-11-14T06:44:00.8941617Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fb3b3aa6d10119b6.xml 2023-11-14T06:44:00.8942949Z ============================= test session starts ============================== 2023-11-14T06:44:00.8944069Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.8944967Z cachedir: .pytest_cache 2023-11-14T06:44:00.8946083Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.8947158Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.8947663Z configfile: pytest.ini 2023-11-14T06:44:00.8948735Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.8949760Z collecting ... collected 1 item 2023-11-14T06:44:00.8950839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2023-11-14T06:44:00.8951769Z 2023-11-14T06:44:00.8953535Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.8955443Z PASSED [5.6105s] [100%] 2023-11-14T06:44:00.8955715Z 2023-11-14T06:44:00.8956965Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fb3b3aa6d10119b6.xml - 2023-11-14T06:44:00.8958644Z ============================== 1 passed in 5.65s =============================== 2023-11-14T06:44:00.8980200Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9995933d20e8dd3.xml 2023-11-14T06:44:00.8981585Z ============================= test session starts ============================== 2023-11-14T06:44:00.8982687Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.8983594Z cachedir: .pytest_cache 2023-11-14T06:44:00.8984722Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.8985817Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.8986309Z configfile: pytest.ini 2023-11-14T06:44:00.8987377Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.8988430Z collecting ... collected 1 item 2023-11-14T06:44:00.8989656Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2023-11-14T06:44:00.8990762Z 2023-11-14T06:44:00.8992726Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.8995172Z [1699942974.347932] [99f90b6e6ff3:9426 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.8996526Z [1699942974.349715] [99f90b6e6ff3:9427 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.8997931Z [1699942974.461550] [99f90b6e6ff3:9427 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.8999470Z [1699942974.461550] [99f90b6e6ff3:9427 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9000989Z [1699942974.459048] [99f90b6e6ff3:9426 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9002722Z [1699942974.459048] [99f90b6e6ff3:9426 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9003726Z PASSED [6.2131s] [100%] 2023-11-14T06:44:00.9004023Z 2023-11-14T06:44:00.9005284Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9995933d20e8dd3.xml - 2023-11-14T06:44:00.9006850Z ============================== 1 passed in 6.25s =============================== 2023-11-14T06:44:00.9008308Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-824eb2f662735eb1.xml 2023-11-14T06:44:00.9009662Z ============================= test session starts ============================== 2023-11-14T06:44:00.9011458Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9012399Z cachedir: .pytest_cache 2023-11-14T06:44:00.9013523Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9014628Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9015140Z configfile: pytest.ini 2023-11-14T06:44:00.9016197Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9017249Z collecting ... collected 1 item 2023-11-14T06:44:00.9018465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2023-11-14T06:44:00.9019541Z 2023-11-14T06:44:00.9021580Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9024151Z [1699942985.889808] [99f90b6e6ff3:9450 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9025680Z [1699942985.889808] [99f90b6e6ff3:9450 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9027200Z [1699942985.880726] [99f90b6e6ff3:9449 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9028712Z [1699942985.880726] [99f90b6e6ff3:9449 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9029714Z PASSED [6.7014s] [100%] 2023-11-14T06:44:00.9030012Z 2023-11-14T06:44:00.9031276Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-824eb2f662735eb1.xml - 2023-11-14T06:44:00.9032833Z ============================== 1 passed in 6.74s =============================== 2023-11-14T06:44:00.9034315Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-251d1c5a3867800e.xml 2023-11-14T06:44:00.9035636Z ============================= test session starts ============================== 2023-11-14T06:44:00.9036750Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9037651Z cachedir: .pytest_cache 2023-11-14T06:44:00.9038757Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9039859Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9040368Z configfile: pytest.ini 2023-11-14T06:44:00.9041424Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9042472Z collecting ... collected 1 item 2023-11-14T06:44:00.9043736Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2023-11-14T06:44:00.9045029Z 2023-11-14T06:44:00.9046984Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9049513Z [1699942997.860159] [99f90b6e6ff3:9477 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9051727Z [1699942997.860159] [99f90b6e6ff3:9477 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9053277Z [1699942997.852905] [99f90b6e6ff3:9476 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9054792Z [1699942997.852905] [99f90b6e6ff3:9476 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9055803Z PASSED [6.2004s] [100%] 2023-11-14T06:44:00.9056101Z 2023-11-14T06:44:00.9057361Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-251d1c5a3867800e.xml - 2023-11-14T06:44:00.9058915Z ============================== 1 passed in 6.24s =============================== 2023-11-14T06:44:00.9060386Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36167b66c6048847.xml 2023-11-14T06:44:00.9061713Z ============================= test session starts ============================== 2023-11-14T06:44:00.9062963Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9063893Z cachedir: .pytest_cache 2023-11-14T06:44:00.9064997Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9066107Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9066616Z configfile: pytest.ini 2023-11-14T06:44:00.9067668Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9068720Z collecting ... collected 1 item 2023-11-14T06:44:00.9069934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2023-11-14T06:44:00.9071004Z 2023-11-14T06:44:00.9072905Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9074884Z PASSED [5.6988s] [100%] 2023-11-14T06:44:00.9075178Z 2023-11-14T06:44:00.9076416Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36167b66c6048847.xml - 2023-11-14T06:44:00.9077974Z ============================== 1 passed in 5.74s =============================== 2023-11-14T06:44:00.9079446Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34238723173fdc59.xml 2023-11-14T06:44:00.9081017Z ============================= test session starts ============================== 2023-11-14T06:44:00.9082173Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9083072Z cachedir: .pytest_cache 2023-11-14T06:44:00.9084184Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9085280Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9085786Z configfile: pytest.ini 2023-11-14T06:44:00.9086854Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9088032Z collecting ... collected 1 item 2023-11-14T06:44:00.9089313Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2023-11-14T06:44:00.9090447Z 2023-11-14T06:44:00.9092604Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9095157Z [1699943020.488680] [99f90b6e6ff3:9516 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9096676Z [1699943020.488680] [99f90b6e6ff3:9516 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9098205Z [1699943020.487556] [99f90b6e6ff3:9515 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9099730Z [1699943020.487556] [99f90b6e6ff3:9515 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9100735Z PASSED [6.4109s] [100%] 2023-11-14T06:44:00.9101031Z 2023-11-14T06:44:00.9102275Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34238723173fdc59.xml - 2023-11-14T06:44:00.9103831Z ============================== 1 passed in 6.45s =============================== 2023-11-14T06:44:00.9105466Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b721d0e9fc0271a6.xml 2023-11-14T06:44:00.9106817Z ============================= test session starts ============================== 2023-11-14T06:44:00.9107933Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9108839Z cachedir: .pytest_cache 2023-11-14T06:44:00.9109961Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9111041Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9111546Z configfile: pytest.ini 2023-11-14T06:44:00.9112617Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9113645Z collecting ... collected 1 item 2023-11-14T06:44:00.9114875Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2023-11-14T06:44:00.9115965Z 2023-11-14T06:44:00.9118168Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (get_future is only supported on mpi, nccl and gloo) [100%] 2023-11-14T06:44:00.9120277Z 2023-11-14T06:44:00.9121546Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b721d0e9fc0271a6.xml - 2023-11-14T06:44:00.9123086Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9124609Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-548bedd17d61563a.xml 2023-11-14T06:44:00.9125954Z ============================= test session starts ============================== 2023-11-14T06:44:00.9127073Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9127958Z cachedir: .pytest_cache 2023-11-14T06:44:00.9129082Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9130324Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9131055Z configfile: pytest.ini 2023-11-14T06:44:00.9132141Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9133191Z collecting ... collected 1 item 2023-11-14T06:44:00.9134467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2023-11-14T06:44:00.9135634Z 2023-11-14T06:44:00.9137882Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (get_future is only supported on mpi, nccl and gloo) [100%] 2023-11-14T06:44:00.9140060Z 2023-11-14T06:44:00.9141314Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-548bedd17d61563a.xml - 2023-11-14T06:44:00.9142871Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9144341Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3939913068963bc6.xml 2023-11-14T06:44:00.9145659Z ============================= test session starts ============================== 2023-11-14T06:44:00.9146778Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9147681Z cachedir: .pytest_cache 2023-11-14T06:44:00.9148912Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9150031Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9150539Z configfile: pytest.ini 2023-11-14T06:44:00.9151626Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9152654Z collecting ... collected 1 item 2023-11-14T06:44:00.9153873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2023-11-14T06:44:00.9154944Z 2023-11-14T06:44:00.9157115Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (get_future is only supported on mpi, nccl and gloo) [100%] 2023-11-14T06:44:00.9159200Z 2023-11-14T06:44:00.9160458Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3939913068963bc6.xml - 2023-11-14T06:44:00.9161985Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9163469Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6fec7e751a09b2b.xml 2023-11-14T06:44:00.9164820Z ============================= test session starts ============================== 2023-11-14T06:44:00.9165934Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9166817Z cachedir: .pytest_cache 2023-11-14T06:44:00.9167937Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9169035Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9169523Z configfile: pytest.ini 2023-11-14T06:44:00.9170599Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9171832Z collecting ... collected 1 item 2023-11-14T06:44:00.9174382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2023-11-14T06:44:00.9175290Z 2023-11-14T06:44:00.9176999Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9179318Z STAGE:2023-11-14 06:24:07 9551:9551 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9180702Z STAGE:2023-11-14 06:24:07 9552:9552 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9182015Z [1699943047.927351] [99f90b6e6ff3:9551 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9183426Z [1699943047.928660] [99f90b6e6ff3:9552 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9184849Z [1699943048.051039] [99f90b6e6ff3:9552 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9186384Z [1699943048.051039] [99f90b6e6ff3:9552 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9187885Z [1699943048.051153] [99f90b6e6ff3:9551 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9189406Z [1699943048.051153] [99f90b6e6ff3:9551 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9191664Z STAGE:2023-11-14 06:24:08 9552:9552 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:24:08 9551:9551 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9193214Z 2023-11-14T06:44:00.9193971Z STAGE:2023-11-14 06:24:08 9552:9552 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9195453Z STAGE:2023-11-14 06:24:08 9551:9551 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9196883Z STAGE:2023-11-14 06:24:08 9552:9552 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9198250Z STAGE:2023-11-14 06:24:08 9551:9551 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9199630Z STAGE:2023-11-14 06:24:08 9552:9552 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9201083Z STAGE:2023-11-14 06:24:08 9552:9552 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9202529Z STAGE:2023-11-14 06:24:08 9551:9551 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9203990Z STAGE:2023-11-14 06:24:08 9551:9551 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9204917Z PASSED [6.3117s] [100%] 2023-11-14T06:44:00.9205215Z 2023-11-14T06:44:00.9206465Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6fec7e751a09b2b.xml - 2023-11-14T06:44:00.9208038Z ============================== 1 passed in 6.35s =============================== 2023-11-14T06:44:00.9209505Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ac7803ab496999b.xml 2023-11-14T06:44:00.9211333Z ============================= test session starts ============================== 2023-11-14T06:44:00.9212604Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9213510Z cachedir: .pytest_cache 2023-11-14T06:44:00.9214626Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9215730Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9216238Z configfile: pytest.ini 2023-11-14T06:44:00.9217461Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9218509Z collecting ... collected 1 item 2023-11-14T06:44:00.9219648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2023-11-14T06:44:00.9220643Z 2023-11-14T06:44:00.9222689Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (ucc does not support all_gather_coalesced) [100%] 2023-11-14T06:44:00.9224716Z 2023-11-14T06:44:00.9225968Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ac7803ab496999b.xml - 2023-11-14T06:44:00.9227526Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9229012Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ac7b47963cd526f.xml 2023-11-14T06:44:00.9230360Z ============================= test session starts ============================== 2023-11-14T06:44:00.9231454Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9232357Z cachedir: .pytest_cache 2023-11-14T06:44:00.9233480Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9234558Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9235068Z configfile: pytest.ini 2023-11-14T06:44:00.9236297Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9237365Z collecting ... collected 1 item 2023-11-14T06:44:00.9238508Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2023-11-14T06:44:00.9239550Z 2023-11-14T06:44:00.9241601Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (ucc does not support all_gather_coalesced) [100%] 2023-11-14T06:44:00.9243608Z 2023-11-14T06:44:00.9244848Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ac7b47963cd526f.xml - 2023-11-14T06:44:00.9246407Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9247872Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a87c8b7c02d77011.xml 2023-11-14T06:44:00.9249223Z ============================= test session starts ============================== 2023-11-14T06:44:00.9250344Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9251442Z cachedir: .pytest_cache 2023-11-14T06:44:00.9252569Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9253668Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9254174Z configfile: pytest.ini 2023-11-14T06:44:00.9255218Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9256263Z collecting ... collected 1 item 2023-11-14T06:44:00.9257395Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2023-11-14T06:44:00.9258380Z 2023-11-14T06:44:00.9260399Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (ucc does not support all_gather_coalesced) [100%] 2023-11-14T06:44:00.9262546Z 2023-11-14T06:44:00.9263796Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a87c8b7c02d77011.xml - 2023-11-14T06:44:00.9265350Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9266825Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-062f354a80c9003d.xml 2023-11-14T06:44:00.9268171Z ============================= test session starts ============================== 2023-11-14T06:44:00.9269268Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9270173Z cachedir: .pytest_cache 2023-11-14T06:44:00.9271297Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9272373Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9272886Z configfile: pytest.ini 2023-11-14T06:44:00.9273954Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9274981Z collecting ... collected 1 item 2023-11-14T06:44:00.9276033Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2023-11-14T06:44:00.9276946Z 2023-11-14T06:44:00.9278969Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all gather) [100%] 2023-11-14T06:44:00.9280832Z 2023-11-14T06:44:00.9282100Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-062f354a80c9003d.xml - 2023-11-14T06:44:00.9283644Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9285123Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-612594180c7a334f.xml 2023-11-14T06:44:00.9286465Z ============================= test session starts ============================== 2023-11-14T06:44:00.9287574Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9288453Z cachedir: .pytest_cache 2023-11-14T06:44:00.9289575Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9290847Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9291357Z configfile: pytest.ini 2023-11-14T06:44:00.9292485Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9293533Z collecting ... collected 1 item 2023-11-14T06:44:00.9294671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2023-11-14T06:44:00.9295707Z 2023-11-14T06:44:00.9297779Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl supports CUDA all_gather_into_tensor) [100%] 2023-11-14T06:44:00.9299807Z 2023-11-14T06:44:00.9301049Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-612594180c7a334f.xml - 2023-11-14T06:44:00.9302596Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9304228Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e15f59b8615e984.xml 2023-11-14T06:44:00.9305551Z ============================= test session starts ============================== 2023-11-14T06:44:00.9306671Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9307571Z cachedir: .pytest_cache 2023-11-14T06:44:00.9308681Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9309778Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9310286Z configfile: pytest.ini 2023-11-14T06:44:00.9311341Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9312395Z collecting ... collected 1 item 2023-11-14T06:44:00.9313566Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2023-11-14T06:44:00.9314600Z 2023-11-14T06:44:00.9316700Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all_gather_into_tensor) [100%] 2023-11-14T06:44:00.9318723Z 2023-11-14T06:44:00.9319981Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e15f59b8615e984.xml - 2023-11-14T06:44:00.9321519Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9323105Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-659f02b19ea47eee.xml 2023-11-14T06:44:00.9324526Z ============================= test session starts ============================== 2023-11-14T06:44:00.9325632Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9326535Z cachedir: .pytest_cache 2023-11-14T06:44:00.9327653Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9328749Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9329238Z configfile: pytest.ini 2023-11-14T06:44:00.9330303Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9331520Z collecting ... collected 1 item 2023-11-14T06:44:00.9332597Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu 2023-11-14T06:44:00.9333564Z 2023-11-14T06:44:00.9335567Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl backend supports allgather multigpu) [100%] 2023-11-14T06:44:00.9337536Z 2023-11-14T06:44:00.9338777Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-659f02b19ea47eee.xml - 2023-11-14T06:44:00.9340339Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9341799Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1850970a7d9f905b.xml 2023-11-14T06:44:00.9343147Z ============================= test session starts ============================== 2023-11-14T06:44:00.9344263Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9345169Z cachedir: .pytest_cache 2023-11-14T06:44:00.9346437Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9347536Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9348043Z configfile: pytest.ini 2023-11-14T06:44:00.9349093Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9350139Z collecting ... collected 1 item 2023-11-14T06:44:00.9351274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2023-11-14T06:44:00.9352269Z 2023-11-14T06:44:00.9354092Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9356425Z [1699943096.533629] [99f90b6e6ff3:9595 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9357774Z [1699943096.553711] [99f90b6e6ff3:9596 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9359194Z [1699943096.644365] [99f90b6e6ff3:9596 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9360713Z [1699943096.644365] [99f90b6e6ff3:9596 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9362214Z [1699943096.640898] [99f90b6e6ff3:9595 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9363851Z [1699943096.640898] [99f90b6e6ff3:9595 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9364894Z PASSED [6.2131s] [100%] 2023-11-14T06:44:00.9365169Z 2023-11-14T06:44:00.9366416Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1850970a7d9f905b.xml - 2023-11-14T06:44:00.9367979Z ============================== 1 passed in 6.25s =============================== 2023-11-14T06:44:00.9369458Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5f5e8cd02421831.xml 2023-11-14T06:44:00.9370969Z ============================= test session starts ============================== 2023-11-14T06:44:00.9372075Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9372977Z cachedir: .pytest_cache 2023-11-14T06:44:00.9374105Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9375192Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9375702Z configfile: pytest.ini 2023-11-14T06:44:00.9376779Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9377836Z collecting ... collected 1 item 2023-11-14T06:44:00.9379002Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2023-11-14T06:44:00.9380060Z 2023-11-14T06:44:00.9382167Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:00.9384218Z 2023-11-14T06:44:00.9385465Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5f5e8cd02421831.xml - 2023-11-14T06:44:00.9387021Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9388690Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ebd9165e16e197d.xml 2023-11-14T06:44:00.9390040Z ============================= test session starts ============================== 2023-11-14T06:44:00.9391156Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9392056Z cachedir: .pytest_cache 2023-11-14T06:44:00.9393163Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9394261Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9394768Z configfile: pytest.ini 2023-11-14T06:44:00.9395825Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9396877Z collecting ... collected 1 item 2023-11-14T06:44:00.9398060Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2023-11-14T06:44:00.9399103Z 2023-11-14T06:44:00.9401197Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:00.9403242Z 2023-11-14T06:44:00.9404486Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ebd9165e16e197d.xml - 2023-11-14T06:44:00.9406042Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9407642Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6abb1bb5b1384f4a.xml 2023-11-14T06:44:00.9409008Z ============================= test session starts ============================== 2023-11-14T06:44:00.9410108Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9411180Z cachedir: .pytest_cache 2023-11-14T06:44:00.9412305Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9413381Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9413892Z configfile: pytest.ini 2023-11-14T06:44:00.9414964Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9415990Z collecting ... collected 1 item 2023-11-14T06:44:00.9417155Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2023-11-14T06:44:00.9418187Z 2023-11-14T06:44:00.9420256Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:00.9422263Z 2023-11-14T06:44:00.9423526Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6abb1bb5b1384f4a.xml - 2023-11-14T06:44:00.9425131Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9426600Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7bd70dacad80222.xml 2023-11-14T06:44:00.9427965Z ============================= test session starts ============================== 2023-11-14T06:44:00.9429083Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9429962Z cachedir: .pytest_cache 2023-11-14T06:44:00.9431280Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9432374Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9432868Z configfile: pytest.ini 2023-11-14T06:44:00.9433935Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9434984Z collecting ... collected 1 item 2023-11-14T06:44:00.9436143Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2023-11-14T06:44:00.9437157Z 2023-11-14T06:44:00.9439233Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:00.9441269Z 2023-11-14T06:44:00.9442514Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7bd70dacad80222.xml - 2023-11-14T06:44:00.9444081Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9445560Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3f3b69ec8160376.xml 2023-11-14T06:44:00.9446890Z ============================= test session starts ============================== 2023-11-14T06:44:00.9448001Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9448901Z cachedir: .pytest_cache 2023-11-14T06:44:00.9450137Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9451419Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9451942Z configfile: pytest.ini 2023-11-14T06:44:00.9453019Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9454045Z collecting ... collected 1 item 2023-11-14T06:44:00.9455227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2023-11-14T06:44:00.9456261Z 2023-11-14T06:44:00.9458387Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:00.9460415Z 2023-11-14T06:44:00.9461686Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3f3b69ec8160376.xml - 2023-11-14T06:44:00.9463228Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9464708Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d0691b5cc05adb7.xml 2023-11-14T06:44:00.9466055Z ============================= test session starts ============================== 2023-11-14T06:44:00.9467165Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9468044Z cachedir: .pytest_cache 2023-11-14T06:44:00.9469166Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9470260Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9470748Z configfile: pytest.ini 2023-11-14T06:44:00.9471821Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9472872Z collecting ... collected 1 item 2023-11-14T06:44:00.9474110Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2023-11-14T06:44:00.9475101Z 2023-11-14T06:44:00.9477142Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:00.9479151Z 2023-11-14T06:44:00.9480394Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d0691b5cc05adb7.xml - 2023-11-14T06:44:00.9481959Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9483446Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9d5dab2ec8bb894.xml 2023-11-14T06:44:00.9484787Z ============================= test session starts ============================== 2023-11-14T06:44:00.9485906Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9486808Z cachedir: .pytest_cache 2023-11-14T06:44:00.9487909Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9489006Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9489514Z configfile: pytest.ini 2023-11-14T06:44:00.9490560Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9491855Z collecting ... collected 1 item 2023-11-14T06:44:00.9493234Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2023-11-14T06:44:00.9494347Z 2023-11-14T06:44:00.9496297Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9498289Z PASSED [5.6152s] [100%] 2023-11-14T06:44:00.9498583Z 2023-11-14T06:44:00.9499833Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9d5dab2ec8bb894.xml - 2023-11-14T06:44:00.9501394Z ============================== 1 passed in 5.65s =============================== 2023-11-14T06:44:00.9502878Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0652134f806cd76d.xml 2023-11-14T06:44:00.9504204Z ============================= test session starts ============================== 2023-11-14T06:44:00.9505320Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9506226Z cachedir: .pytest_cache 2023-11-14T06:44:00.9507329Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9508429Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9508934Z configfile: pytest.ini 2023-11-14T06:44:00.9510003Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9511031Z collecting ... collected 1 item 2023-11-14T06:44:00.9512208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2023-11-14T06:44:00.9513509Z 2023-11-14T06:44:00.9515388Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9517502Z PASSED [5.5990s] [100%] 2023-11-14T06:44:00.9517777Z 2023-11-14T06:44:00.9519024Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0652134f806cd76d.xml - 2023-11-14T06:44:00.9520579Z ============================== 1 passed in 5.64s =============================== 2023-11-14T06:44:00.9522058Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7dec3dd8567cb251.xml 2023-11-14T06:44:00.9523384Z ============================= test session starts ============================== 2023-11-14T06:44:00.9524561Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9525461Z cachedir: .pytest_cache 2023-11-14T06:44:00.9526589Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9527675Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9528181Z configfile: pytest.ini 2023-11-14T06:44:00.9529251Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9530280Z collecting ... collected 1 item 2023-11-14T06:44:00.9531647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2023-11-14T06:44:00.9532634Z 2023-11-14T06:44:00.9534566Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9536988Z STAGE:2023-11-14 06:26:01 9658:9658 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9538343Z STAGE:2023-11-14 06:26:01 9657:9657 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9539686Z [1699943161.582577] [99f90b6e6ff3:9657 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9541038Z [1699943161.583848] [99f90b6e6ff3:9658 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9542446Z [1699943161.702852] [99f90b6e6ff3:9658 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9543974Z [1699943161.702852] [99f90b6e6ff3:9658 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9545501Z [1699943161.702255] [99f90b6e6ff3:9657 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9547020Z [1699943161.702255] [99f90b6e6ff3:9657 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9549257Z STAGE:2023-11-14 06:26:02 9658:9658 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:26:02 9657:9657 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9550662Z 2023-11-14T06:44:00.9551361Z STAGE:2023-11-14 06:26:02 9658:9658 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9552855Z STAGE:2023-11-14 06:26:02 9657:9657 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9554279Z STAGE:2023-11-14 06:26:02 9657:9657 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9555624Z STAGE:2023-11-14 06:26:02 9658:9658 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9557017Z STAGE:2023-11-14 06:26:02 9657:9657 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9558427Z STAGE:2023-11-14 06:26:02 9658:9658 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9560140Z STAGE:2023-11-14 06:26:02 9657:9657 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9561601Z STAGE:2023-11-14 06:26:02 9658:9658 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9562586Z PASSED [6.3124s] [100%] 2023-11-14T06:44:00.9562862Z 2023-11-14T06:44:00.9564127Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7dec3dd8567cb251.xml - 2023-11-14T06:44:00.9565664Z ============================== 1 passed in 6.35s =============================== 2023-11-14T06:44:00.9567142Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b0137aff45ab4a9.xml 2023-11-14T06:44:00.9568494Z ============================= test session starts ============================== 2023-11-14T06:44:00.9569607Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9570496Z cachedir: .pytest_cache 2023-11-14T06:44:00.9571859Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9572963Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9573453Z configfile: pytest.ini 2023-11-14T06:44:00.9574520Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9575567Z collecting ... collected 1 item 2023-11-14T06:44:00.9576670Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2023-11-14T06:44:00.9577666Z 2023-11-14T06:44:00.9579569Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9581997Z STAGE:2023-11-14 06:26:13 9680:9680 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9583372Z STAGE:2023-11-14 06:26:13 9681:9681 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9584675Z [1699943173.333769] [99f90b6e6ff3:9680 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9586029Z [1699943173.339935] [99f90b6e6ff3:9681 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9587454Z [1699943173.457804] [99f90b6e6ff3:9681 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9588987Z [1699943173.457804] [99f90b6e6ff3:9681 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9590484Z [1699943173.456417] [99f90b6e6ff3:9680 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9592067Z [1699943173.456417] [99f90b6e6ff3:9680 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9594321Z STAGE:2023-11-14 06:26:13 9681:9681 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:26:13 9680:9680 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9595704Z 2023-11-14T06:44:00.9596427Z STAGE:2023-11-14 06:26:13 9681:9681 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9597900Z STAGE:2023-11-14 06:26:13 9680:9680 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9599331Z STAGE:2023-11-14 06:26:14 9681:9681 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9600704Z STAGE:2023-11-14 06:26:14 9680:9680 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9602103Z STAGE:2023-11-14 06:26:14 9681:9681 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9603631Z STAGE:2023-11-14 06:26:14 9680:9680 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9605084Z STAGE:2023-11-14 06:26:14 9681:9681 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9606568Z STAGE:2023-11-14 06:26:14 9680:9680 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9607488Z PASSED [6.4122s] [100%] 2023-11-14T06:44:00.9607780Z 2023-11-14T06:44:00.9609027Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b0137aff45ab4a9.xml - 2023-11-14T06:44:00.9610590Z ============================== 1 passed in 6.45s =============================== 2023-11-14T06:44:00.9612300Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e242eef12462a35.xml 2023-11-14T06:44:00.9613644Z ============================= test session starts ============================== 2023-11-14T06:44:00.9614764Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9615666Z cachedir: .pytest_cache 2023-11-14T06:44:00.9616771Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9617871Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9618382Z configfile: pytest.ini 2023-11-14T06:44:00.9619437Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9620485Z collecting ... collected 1 item 2023-11-14T06:44:00.9621731Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2023-11-14T06:44:00.9622731Z 2023-11-14T06:44:00.9624587Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9626970Z STAGE:2023-11-14 06:26:25 9704:9704 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9628343Z STAGE:2023-11-14 06:26:25 9703:9703 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9629663Z [1699943185.121869] [99f90b6e6ff3:9703 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9631015Z [1699943185.127870] [99f90b6e6ff3:9704 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9632421Z [1699943185.246881] [99f90b6e6ff3:9704 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9633952Z [1699943185.246881] [99f90b6e6ff3:9704 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9635500Z STAGE:2023-11-14 06:26:25 9704:9704 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9636917Z [1699943185.244633] [99f90b6e6ff3:9703 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9638421Z [1699943185.244633] [99f90b6e6ff3:9703 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9639956Z STAGE:2023-11-14 06:26:25 9703:9703 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9641417Z STAGE:2023-11-14 06:26:25 9704:9704 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9642894Z STAGE:2023-11-14 06:26:25 9703:9703 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9644313Z STAGE:2023-11-14 06:26:25 9704:9704 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9645836Z STAGE:2023-11-14 06:26:25 9703:9703 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9647229Z STAGE:2023-11-14 06:26:25 9704:9704 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9648616Z STAGE:2023-11-14 06:26:25 9703:9703 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9650071Z STAGE:2023-11-14 06:26:25 9704:9704 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9651805Z STAGE:2023-11-14 06:26:25 9703:9703 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9652725Z PASSED [6.4070s] [100%] 2023-11-14T06:44:00.9653018Z 2023-11-14T06:44:00.9654274Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e242eef12462a35.xml - 2023-11-14T06:44:00.9655827Z ============================== 1 passed in 6.44s =============================== 2023-11-14T06:44:00.9657308Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-636b5a49a8116603.xml 2023-11-14T06:44:00.9658628Z ============================= test session starts ============================== 2023-11-14T06:44:00.9659741Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9660640Z cachedir: .pytest_cache 2023-11-14T06:44:00.9661745Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9662845Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9663350Z configfile: pytest.ini 2023-11-14T06:44:00.9664544Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9665588Z collecting ... collected 1 item 2023-11-14T06:44:00.9666682Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2023-11-14T06:44:00.9667634Z 2023-11-14T06:44:00.9669403Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9671440Z SKIPPED [5.5124s] (Skipped due to small world size.) [100%] 2023-11-14T06:44:00.9671937Z 2023-11-14T06:44:00.9673187Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-636b5a49a8116603.xml - 2023-11-14T06:44:00.9674754Z ============================== 1 skipped in 5.55s ============================== 2023-11-14T06:44:00.9676234Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55e11960bd7b413b.xml 2023-11-14T06:44:00.9677568Z ============================= test session starts ============================== 2023-11-14T06:44:00.9678684Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9679590Z cachedir: .pytest_cache 2023-11-14T06:44:00.9680714Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9681795Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9682302Z configfile: pytest.ini 2023-11-14T06:44:00.9683370Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9684401Z collecting ... collected 1 item 2023-11-14T06:44:00.9685521Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2023-11-14T06:44:00.9686493Z 2023-11-14T06:44:00.9688284Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9690470Z SKIPPED [5.5059s] (Skipped due to small world size.) [100%] 2023-11-14T06:44:00.9691190Z 2023-11-14T06:44:00.9692453Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55e11960bd7b413b.xml - 2023-11-14T06:44:00.9694011Z ============================== 1 skipped in 5.54s ============================== 2023-11-14T06:44:00.9695492Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66611a977788ea0e.xml 2023-11-14T06:44:00.9696842Z ============================= test session starts ============================== 2023-11-14T06:44:00.9697939Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9698849Z cachedir: .pytest_cache 2023-11-14T06:44:00.9699968Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9701044Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9701554Z configfile: pytest.ini 2023-11-14T06:44:00.9702622Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9703649Z collecting ... collected 1 item 2023-11-14T06:44:00.9704698Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2023-11-14T06:44:00.9705604Z 2023-11-14T06:44:00.9707451Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9709818Z STAGE:2023-11-14 06:26:58 9751:9751 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9711179Z STAGE:2023-11-14 06:26:58 9750:9750 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9712501Z [1699943218.629760] [99f90b6e6ff3:9750 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9713856Z [1699943218.630175] [99f90b6e6ff3:9751 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9715282Z [1699943218.751003] [99f90b6e6ff3:9751 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9716796Z [1699943218.751003] [99f90b6e6ff3:9751 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9718321Z [1699943218.749709] [99f90b6e6ff3:9750 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9719841Z [1699943218.749709] [99f90b6e6ff3:9750 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9722100Z STAGE:2023-11-14 06:26:59 9751:9751 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:26:59 9750:9750 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9723478Z 2023-11-14T06:44:00.9724973Z STAGE:2023-11-14 06:26:59 9751:9751 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:26:59 9750:9750 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9726443Z 2023-11-14T06:44:00.9727082Z STAGE:2023-11-14 06:26:59 9751:9751 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9728459Z STAGE:2023-11-14 06:26:59 9750:9750 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9729851Z STAGE:2023-11-14 06:26:59 9751:9751 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9732385Z STAGE:2023-11-14 06:26:59 9751:9751 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:26:59 9750:9750 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9733825Z 2023-11-14T06:44:00.9734521Z STAGE:2023-11-14 06:26:59 9750:9750 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9735465Z PASSED [6.2993s] [100%] 2023-11-14T06:44:00.9735742Z 2023-11-14T06:44:00.9737005Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66611a977788ea0e.xml - 2023-11-14T06:44:00.9738540Z ============================== 1 passed in 6.34s =============================== 2023-11-14T06:44:00.9740020Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18251fa92968b03b.xml 2023-11-14T06:44:00.9741372Z ============================= test session starts ============================== 2023-11-14T06:44:00.9742467Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9743367Z cachedir: .pytest_cache 2023-11-14T06:44:00.9744485Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9745583Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9746071Z configfile: pytest.ini 2023-11-14T06:44:00.9747145Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9748194Z collecting ... collected 1 item 2023-11-14T06:44:00.9749392Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2023-11-14T06:44:00.9750385Z 2023-11-14T06:44:00.9752139Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9754517Z STAGE:2023-11-14 06:27:10 9774:9774 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9755897Z STAGE:2023-11-14 06:27:10 9773:9773 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9757198Z [1699943230.346036] [99f90b6e6ff3:9773 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9758544Z [1699943230.366026] [99f90b6e6ff3:9774 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9759971Z [1699943230.470444] [99f90b6e6ff3:9774 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9761501Z [1699943230.470444] [99f90b6e6ff3:9774 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9763006Z [1699943230.464519] [99f90b6e6ff3:9773 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9764532Z [1699943230.464519] [99f90b6e6ff3:9773 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9766784Z STAGE:2023-11-14 06:27:10 9774:9774 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:27:10 9773:9773 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9768168Z 2023-11-14T06:44:00.9768882Z STAGE:2023-11-14 06:27:10 9774:9774 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9770355Z STAGE:2023-11-14 06:27:10 9773:9773 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9772264Z STAGE:2023-11-14 06:27:11 9773:9773 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9773837Z STAGE:2023-11-14 06:27:11 9774:9774 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9775206Z STAGE:2023-11-14 06:27:11 9773:9773 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9776615Z STAGE:2023-11-14 06:27:11 9774:9774 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9778069Z STAGE:2023-11-14 06:27:11 9773:9773 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9779548Z STAGE:2023-11-14 06:27:11 9774:9774 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9780465Z PASSED [6.4014s] [100%] 2023-11-14T06:44:00.9780761Z 2023-11-14T06:44:00.9782007Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18251fa92968b03b.xml - 2023-11-14T06:44:00.9783560Z ============================== 1 passed in 6.44s =============================== 2023-11-14T06:44:00.9785018Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7f3c6a866847600.xml 2023-11-14T06:44:00.9786364Z ============================= test session starts ============================== 2023-11-14T06:44:00.9787484Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9788385Z cachedir: .pytest_cache 2023-11-14T06:44:00.9789485Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9790588Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9791093Z configfile: pytest.ini 2023-11-14T06:44:00.9792281Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9793349Z collecting ... collected 1 item 2023-11-14T06:44:00.9794456Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2023-11-14T06:44:00.9795427Z 2023-11-14T06:44:00.9797210Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:00.9799573Z STAGE:2023-11-14 06:27:21 9796:9796 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9800955Z STAGE:2023-11-14 06:27:22 9797:9797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9802282Z [1699943242.054103] [99f90b6e6ff3:9796 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9803627Z [1699943242.056023] [99f90b6e6ff3:9797 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:00.9805049Z [1699943242.178686] [99f90b6e6ff3:9797 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9806588Z [1699943242.178686] [99f90b6e6ff3:9797 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9808107Z [1699943242.178764] [99f90b6e6ff3:9796 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:00.9809602Z [1699943242.178764] [99f90b6e6ff3:9796 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:00.9812100Z STAGE:2023-11-14 06:27:22 9797:9797 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:27:22 9796:9796 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9813508Z 2023-11-14T06:44:00.9814963Z STAGE:2023-11-14 06:27:22 9797:9797 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:27:22 9796:9796 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9816559Z 2023-11-14T06:44:00.9817225Z STAGE:2023-11-14 06:27:22 9797:9797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9818596Z STAGE:2023-11-14 06:27:22 9796:9796 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:00.9819971Z STAGE:2023-11-14 06:27:22 9797:9797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9821389Z STAGE:2023-11-14 06:27:22 9796:9796 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:00.9822837Z STAGE:2023-11-14 06:27:22 9797:9797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9824358Z STAGE:2023-11-14 06:27:22 9796:9796 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:00.9825303Z PASSED [6.4120s] [100%] 2023-11-14T06:44:00.9825580Z 2023-11-14T06:44:00.9826843Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7f3c6a866847600.xml - 2023-11-14T06:44:00.9828406Z ============================== 1 passed in 6.45s =============================== 2023-11-14T06:44:00.9829865Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-82d51c3b0a42ebe2.xml 2023-11-14T06:44:00.9831212Z ============================= test session starts ============================== 2023-11-14T06:44:00.9832329Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9833209Z cachedir: .pytest_cache 2023-11-14T06:44:00.9834331Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9835542Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9836049Z configfile: pytest.ini 2023-11-14T06:44:00.9837119Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9838170Z collecting ... collected 1 item 2023-11-14T06:44:00.9839290Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2023-11-14T06:44:00.9840263Z 2023-11-14T06:44:00.9842386Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Gloo and NCCL backends will have CUDA allReduce tested) [100%] 2023-11-14T06:44:00.9844464Z 2023-11-14T06:44:00.9845715Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-82d51c3b0a42ebe2.xml - 2023-11-14T06:44:00.9847280Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9848767Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f31500b27f604fb6.xml 2023-11-14T06:44:00.9850092Z ============================= test session starts ============================== 2023-11-14T06:44:00.9851437Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9852346Z cachedir: .pytest_cache 2023-11-14T06:44:00.9853466Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9854540Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9855050Z configfile: pytest.ini 2023-11-14T06:44:00.9856124Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9857149Z collecting ... collected 1 item 2023-11-14T06:44:00.9858290Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2023-11-14T06:44:00.9859421Z 2023-11-14T06:44:00.9861583Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Gloo and NCCL backends will have CUDA allReduce tested) [100%] 2023-11-14T06:44:00.9863654Z 2023-11-14T06:44:00.9864914Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f31500b27f604fb6.xml - 2023-11-14T06:44:00.9866456Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9867945Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0b38e067c3a046c.xml 2023-11-14T06:44:00.9869298Z ============================= test session starts ============================== 2023-11-14T06:44:00.9870418Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9871297Z cachedir: .pytest_cache 2023-11-14T06:44:00.9872419Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9873526Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9874015Z configfile: pytest.ini 2023-11-14T06:44:00.9875082Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9876131Z collecting ... collected 1 item 2023-11-14T06:44:00.9877294Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2023-11-14T06:44:00.9878261Z 2023-11-14T06:44:00.9880132Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only MPI supports all_to_all) [100%] 2023-11-14T06:44:00.9881982Z 2023-11-14T06:44:00.9883220Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0b38e067c3a046c.xml - 2023-11-14T06:44:00.9884777Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9886247Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4134150f7599783.xml 2023-11-14T06:44:00.9887574Z ============================= test session starts ============================== 2023-11-14T06:44:00.9888698Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9889598Z cachedir: .pytest_cache 2023-11-14T06:44:00.9890920Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9892096Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9892610Z configfile: pytest.ini 2023-11-14T06:44:00.9893671Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9894724Z collecting ... collected 1 item 2023-11-14T06:44:00.9895775Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2023-11-14T06:44:00.9896686Z 2023-11-14T06:44:00.9898604Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only NCCL supports CUDA all_to_all) [100%] 2023-11-14T06:44:00.9900449Z 2023-11-14T06:44:00.9901709Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4134150f7599783.xml - 2023-11-14T06:44:00.9903386Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9904873Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-67b472c223c06aa6.xml 2023-11-14T06:44:00.9906220Z ============================= test session starts ============================== 2023-11-14T06:44:00.9907315Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9908216Z cachedir: .pytest_cache 2023-11-14T06:44:00.9909339Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9910445Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9910935Z configfile: pytest.ini 2023-11-14T06:44:00.9912001Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9913052Z collecting ... collected 1 item 2023-11-14T06:44:00.9914139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2023-11-14T06:44:00.9915131Z 2023-11-14T06:44:00.9917060Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only NCCL supports CUDA all_to_all) [100%] 2023-11-14T06:44:00.9918967Z 2023-11-14T06:44:00.9920337Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-67b472c223c06aa6.xml - 2023-11-14T06:44:00.9921914Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9923373Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3cc031cfd9988cef.xml 2023-11-14T06:44:00.9924779Z ============================= test session starts ============================== 2023-11-14T06:44:00.9925896Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9926794Z cachedir: .pytest_cache 2023-11-14T06:44:00.9927895Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9928995Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9929503Z configfile: pytest.ini 2023-11-14T06:44:00.9930555Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9931853Z collecting ... collected 1 item 2023-11-14T06:44:00.9932979Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2023-11-14T06:44:00.9933969Z 2023-11-14T06:44:00.9935924Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only NCCL supports CUDA all_to_all) [100%] 2023-11-14T06:44:00.9937839Z 2023-11-14T06:44:00.9939079Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3cc031cfd9988cef.xml - 2023-11-14T06:44:00.9940637Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9942126Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4b221e1e2ef3476.xml 2023-11-14T06:44:00.9943477Z ============================= test session starts ============================== 2023-11-14T06:44:00.9944749Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9945648Z cachedir: .pytest_cache 2023-11-14T06:44:00.9946772Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9947852Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9948362Z configfile: pytest.ini 2023-11-14T06:44:00.9949430Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9950456Z collecting ... collected 1 item 2023-11-14T06:44:00.9951549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2023-11-14T06:44:00.9952521Z 2023-11-14T06:44:00.9954503Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all_to_all_single) [100%] 2023-11-14T06:44:00.9956436Z 2023-11-14T06:44:00.9957702Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4b221e1e2ef3476.xml - 2023-11-14T06:44:00.9959263Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9960721Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca29b72246906cad.xml 2023-11-14T06:44:00.9962069Z ============================= test session starts ============================== 2023-11-14T06:44:00.9963299Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9964194Z cachedir: .pytest_cache 2023-11-14T06:44:00.9965320Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9966423Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9966909Z configfile: pytest.ini 2023-11-14T06:44:00.9967976Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9969022Z collecting ... collected 1 item 2023-11-14T06:44:00.9970168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2023-11-14T06:44:00.9971496Z 2023-11-14T06:44:00.9973532Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only MPI supports CPU all_to_all_single) [100%] 2023-11-14T06:44:00.9975502Z 2023-11-14T06:44:00.9976749Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca29b72246906cad.xml - 2023-11-14T06:44:00.9978312Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9979790Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3151c65c938afa9a.xml 2023-11-14T06:44:00.9981111Z ============================= test session starts ============================== 2023-11-14T06:44:00.9982219Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:00.9983119Z cachedir: .pytest_cache 2023-11-14T06:44:00.9984221Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:00.9985329Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:00.9985842Z configfile: pytest.ini 2023-11-14T06:44:00.9986912Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:00.9988086Z collecting ... collected 1 item 2023-11-14T06:44:00.9989276Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2023-11-14T06:44:00.9990322Z 2023-11-14T06:44:00.9992401Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only MPI supports CPU all_to_all_single) [100%] 2023-11-14T06:44:00.9994390Z 2023-11-14T06:44:00.9995659Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3151c65c938afa9a.xml - 2023-11-14T06:44:00.9997199Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:00.9998696Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b042eb72f8d0d934.xml 2023-11-14T06:44:01.0000043Z ============================= test session starts ============================== 2023-11-14T06:44:01.0001150Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0002034Z cachedir: .pytest_cache 2023-11-14T06:44:01.0003155Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0004252Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0004740Z configfile: pytest.ini 2023-11-14T06:44:01.0005922Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0006985Z collecting ... collected 1 item 2023-11-14T06:44:01.0008142Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2023-11-14T06:44:01.0009193Z 2023-11-14T06:44:01.0011483Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all_to_all_single) [100%] 2023-11-14T06:44:01.0013509Z 2023-11-14T06:44:01.0014762Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b042eb72f8d0d934.xml - 2023-11-14T06:44:01.0016320Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0017806Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-123554d2eadf8a9c.xml 2023-11-14T06:44:01.0019140Z ============================= test session starts ============================== 2023-11-14T06:44:01.0020262Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0021166Z cachedir: .pytest_cache 2023-11-14T06:44:01.0022268Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0023372Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0023882Z configfile: pytest.ini 2023-11-14T06:44:01.0024976Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0026033Z collecting ... collected 1 item 2023-11-14T06:44:01.0027262Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2023-11-14T06:44:01.0028341Z 2023-11-14T06:44:01.0030461Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all_to_all_single) [100%] 2023-11-14T06:44:01.0032652Z 2023-11-14T06:44:01.0033918Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-123554d2eadf8a9c.xml - 2023-11-14T06:44:01.0035455Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0036930Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c6ce8193f12614c.xml 2023-11-14T06:44:01.0038275Z ============================= test session starts ============================== 2023-11-14T06:44:01.0039378Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0040282Z cachedir: .pytest_cache 2023-11-14T06:44:01.0041415Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0042515Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0043005Z configfile: pytest.ini 2023-11-14T06:44:01.0044079Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0045130Z collecting ... collected 1 item 2023-11-14T06:44:01.0046349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2023-11-14T06:44:01.0047472Z 2023-11-14T06:44:01.0049702Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all_to_all_single) [100%] 2023-11-14T06:44:01.0052042Z 2023-11-14T06:44:01.0053304Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c6ce8193f12614c.xml - 2023-11-14T06:44:01.0054861Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0056322Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-870bafb679495005.xml 2023-11-14T06:44:01.0057672Z ============================= test session starts ============================== 2023-11-14T06:44:01.0058790Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0059692Z cachedir: .pytest_cache 2023-11-14T06:44:01.0060800Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0061901Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0062416Z configfile: pytest.ini 2023-11-14T06:44:01.0063465Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0064512Z collecting ... collected 1 item 2023-11-14T06:44:01.0065694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2023-11-14T06:44:01.0066732Z 2023-11-14T06:44:01.0068797Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only MPI supports CPU all_to_all_single) [100%] 2023-11-14T06:44:01.0070780Z 2023-11-14T06:44:01.0072022Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-870bafb679495005.xml - 2023-11-14T06:44:01.0073716Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0075196Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c399cafc4c7b443.xml 2023-11-14T06:44:01.0076547Z ============================= test session starts ============================== 2023-11-14T06:44:01.0077640Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0078546Z cachedir: .pytest_cache 2023-11-14T06:44:01.0079665Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0080748Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0081259Z configfile: pytest.ini 2023-11-14T06:44:01.0082333Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0083366Z collecting ... collected 1 item 2023-11-14T06:44:01.0084583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2023-11-14T06:44:01.0085666Z 2023-11-14T06:44:01.0087754Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl supports CUDA all_to_all_single) [100%] 2023-11-14T06:44:01.0089774Z 2023-11-14T06:44:01.0091266Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c399cafc4c7b443.xml - 2023-11-14T06:44:01.0093004Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0094507Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdad373f46edf1b2.xml 2023-11-14T06:44:01.0095868Z ============================= test session starts ============================== 2023-11-14T06:44:01.0096983Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0097866Z cachedir: .pytest_cache 2023-11-14T06:44:01.0098983Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0100078Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0100584Z configfile: pytest.ini 2023-11-14T06:44:01.0101631Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0102676Z collecting ... collected 1 item 2023-11-14T06:44:01.0103867Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2023-11-14T06:44:01.0104913Z 2023-11-14T06:44:01.0106976Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all_to_all_single) [100%] 2023-11-14T06:44:01.0109001Z 2023-11-14T06:44:01.0110253Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdad373f46edf1b2.xml - 2023-11-14T06:44:01.0111815Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0113287Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9693950f331f0f8.xml 2023-11-14T06:44:01.0114618Z ============================= test session starts ============================== 2023-11-14T06:44:01.0115729Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0116772Z cachedir: .pytest_cache 2023-11-14T06:44:01.0117896Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0118971Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0119481Z configfile: pytest.ini 2023-11-14T06:44:01.0120545Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0121575Z collecting ... collected 1 item 2023-11-14T06:44:01.0122814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2023-11-14T06:44:01.0123900Z 2023-11-14T06:44:01.0126083Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all_to_all_single) [100%] 2023-11-14T06:44:01.0128134Z 2023-11-14T06:44:01.0129400Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9693950f331f0f8.xml - 2023-11-14T06:44:01.0131175Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0132666Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da438cc39f1aedac.xml 2023-11-14T06:44:01.0134014Z ============================= test session starts ============================== 2023-11-14T06:44:01.0135253Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0136152Z cachedir: .pytest_cache 2023-11-14T06:44:01.0137280Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0138385Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0138874Z configfile: pytest.ini 2023-11-14T06:44:01.0139941Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0140983Z collecting ... collected 1 item 2023-11-14T06:44:01.0142189Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2023-11-14T06:44:01.0143287Z 2023-11-14T06:44:01.0145381Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only MPI supports CPU all_to_all_single) [100%] 2023-11-14T06:44:01.0147427Z 2023-11-14T06:44:01.0148673Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da438cc39f1aedac.xml - 2023-11-14T06:44:01.0150245Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0151724Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65666c35b2e3b527.xml 2023-11-14T06:44:01.0153046Z ============================= test session starts ============================== 2023-11-14T06:44:01.0154160Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0155062Z cachedir: .pytest_cache 2023-11-14T06:44:01.0156174Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0157272Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0157782Z configfile: pytest.ini 2023-11-14T06:44:01.0158851Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0160014Z collecting ... collected 1 item 2023-11-14T06:44:01.0161086Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2023-11-14T06:44:01.0162020Z 2023-11-14T06:44:01.0163784Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.0166166Z [1699943343.505583] [99f90b6e6ff3:9870 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0167676Z [1699943343.505583] [99f90b6e6ff3:9870 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0169198Z [1699943343.508718] [99f90b6e6ff3:9871 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0170932Z [1699943343.508718] [99f90b6e6ff3:9871 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0171955Z PASSED [7.0076s] [100%] 2023-11-14T06:44:01.0172248Z 2023-11-14T06:44:01.0173492Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65666c35b2e3b527.xml - 2023-11-14T06:44:01.0175051Z ============================== 1 passed in 7.04s =============================== 2023-11-14T06:44:01.0176527Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-363acba8dd921d5e.xml 2023-11-14T06:44:01.0177976Z ============================= test session starts ============================== 2023-11-14T06:44:01.0179118Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0180027Z cachedir: .pytest_cache 2023-11-14T06:44:01.0181129Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0182224Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0182731Z configfile: pytest.ini 2023-11-14T06:44:01.0183798Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0184828Z collecting ... collected 1 item 2023-11-14T06:44:01.0185865Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2023-11-14T06:44:01.0186767Z 2023-11-14T06:44:01.0188640Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires world size of 3) [100%] 2023-11-14T06:44:01.0190461Z 2023-11-14T06:44:01.0191723Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-363acba8dd921d5e.xml - 2023-11-14T06:44:01.0193324Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0194811Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9aaebe9b5bfa90ac.xml 2023-11-14T06:44:01.0196164Z ============================= test session starts ============================== 2023-11-14T06:44:01.0197257Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0198161Z cachedir: .pytest_cache 2023-11-14T06:44:01.0199284Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0200378Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0201004Z configfile: pytest.ini 2023-11-14T06:44:01.0202073Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0203117Z collecting ... collected 1 item 2023-11-14T06:44:01.0204137Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2023-11-14T06:44:01.0205052Z 2023-11-14T06:44:01.0206743Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.0209085Z [1699943361.081530] [99f90b6e6ff3:9906 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0210608Z [1699943361.081530] [99f90b6e6ff3:9906 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0212376Z [1699943361.085981] [99f90b6e6ff3:9907 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0213892Z [1699943361.085981] [99f90b6e6ff3:9907 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0214911Z PASSED [7.0059s] [100%] 2023-11-14T06:44:01.0215181Z 2023-11-14T06:44:01.0216440Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9aaebe9b5bfa90ac.xml - 2023-11-14T06:44:01.0217991Z ============================== 1 passed in 7.04s =============================== 2023-11-14T06:44:01.0219609Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-825f40db14c5ef33.xml 2023-11-14T06:44:01.0220974Z ============================= test session starts ============================== 2023-11-14T06:44:01.0222077Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0222968Z cachedir: .pytest_cache 2023-11-14T06:44:01.0224131Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0225210Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0225709Z configfile: pytest.ini 2023-11-14T06:44:01.0226776Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0227792Z collecting ... collected 1 item 2023-11-14T06:44:01.0228862Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2023-11-14T06:44:01.0229814Z 2023-11-14T06:44:01.0231726Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (ucc does not support CPU barrier) [100%] 2023-11-14T06:44:01.0233588Z 2023-11-14T06:44:01.0234848Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-825f40db14c5ef33.xml - 2023-11-14T06:44:01.0236395Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0237406Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-203fbfb483a08cdd.xml 2023-11-14T06:44:01.0237674Z ============================= test session starts ============================== 2023-11-14T06:44:01.0238338Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0238494Z cachedir: .pytest_cache 2023-11-14T06:44:01.0239334Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0239676Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0239828Z configfile: pytest.ini 2023-11-14T06:44:01.0240637Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0240823Z collecting ... collected 1 item 2023-11-14T06:44:01.0241628Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2023-11-14T06:44:01.0241649Z 2023-11-14T06:44:01.0243637Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only gloo backend supports timeouts) [100%] 2023-11-14T06:44:01.0243661Z 2023-11-14T06:44:01.0244904Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-203fbfb483a08cdd.xml - 2023-11-14T06:44:01.0245192Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0246200Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bcbb976010d37e2.xml 2023-11-14T06:44:01.0246472Z ============================= test session starts ============================== 2023-11-14T06:44:01.0247129Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0247283Z cachedir: .pytest_cache 2023-11-14T06:44:01.0248127Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0248415Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0248582Z configfile: pytest.ini 2023-11-14T06:44:01.0249402Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0249578Z collecting ... collected 1 item 2023-11-14T06:44:01.0250350Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2023-11-14T06:44:01.0250371Z 2023-11-14T06:44:01.0252570Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only gloo backend supports timeouts) [100%] 2023-11-14T06:44:01.0252595Z 2023-11-14T06:44:01.0253850Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bcbb976010d37e2.xml - 2023-11-14T06:44:01.0254141Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0255158Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a95c79ebcc86710f.xml 2023-11-14T06:44:01.0255433Z ============================= test session starts ============================== 2023-11-14T06:44:01.0256090Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0256245Z cachedir: .pytest_cache 2023-11-14T06:44:01.0257087Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0257282Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0257436Z configfile: pytest.ini 2023-11-14T06:44:01.0258250Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0258424Z collecting ... collected 1 item 2023-11-14T06:44:01.0259207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2023-11-14T06:44:01.0259352Z 2023-11-14T06:44:01.0261269Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (NCCL Batch Send Recv Only) [100%] 2023-11-14T06:44:01.0261290Z 2023-11-14T06:44:01.0262529Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a95c79ebcc86710f.xml - 2023-11-14T06:44:01.0262809Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0263820Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-233d4b486a3bcbff.xml 2023-11-14T06:44:01.0264092Z ============================= test session starts ============================== 2023-11-14T06:44:01.0264747Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0264911Z cachedir: .pytest_cache 2023-11-14T06:44:01.0265773Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0265948Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0266102Z configfile: pytest.ini 2023-11-14T06:44:01.0266913Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0267082Z collecting ... collected 1 item 2023-11-14T06:44:01.0267892Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2023-11-14T06:44:01.0267913Z 2023-11-14T06:44:01.0269951Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (NCCL Batch Send Recv Only) [100%] 2023-11-14T06:44:01.0269981Z 2023-11-14T06:44:01.0271252Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-233d4b486a3bcbff.xml - 2023-11-14T06:44:01.0271535Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0272544Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-33c671fc704ea161.xml 2023-11-14T06:44:01.0272810Z ============================= test session starts ============================== 2023-11-14T06:44:01.0273470Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0273632Z cachedir: .pytest_cache 2023-11-14T06:44:01.0274496Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0274679Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0274836Z configfile: pytest.ini 2023-11-14T06:44:01.0275652Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0275822Z collecting ... collected 1 item 2023-11-14T06:44:01.0276679Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2023-11-14T06:44:01.0276700Z 2023-11-14T06:44:01.0278677Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (NCCL Batch Send Recv Only) [100%] 2023-11-14T06:44:01.0278698Z 2023-11-14T06:44:01.0279948Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-33c671fc704ea161.xml - 2023-11-14T06:44:01.0280331Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.0281353Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ad2f3b5b66e86a6.xml 2023-11-14T06:44:01.0281639Z ============================= test session starts ============================== 2023-11-14T06:44:01.0282281Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0282437Z cachedir: .pytest_cache 2023-11-14T06:44:01.0283296Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0283479Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0283632Z configfile: pytest.ini 2023-11-14T06:44:01.0284446Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0284621Z collecting ... collected 1 item 2023-11-14T06:44:01.0285303Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2023-11-14T06:44:01.0285342Z 2023-11-14T06:44:01.0287020Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.0287661Z STAGE:2023-11-14 06:30:04 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0288311Z STAGE:2023-11-14 06:30:04 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0288986Z [1699943404.893644] [99f90b6e6ff3:9947 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0289577Z [1699943404.899505] [99f90b6e6ff3:9948 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0290254Z [1699943405.013859] [99f90b6e6ff3:9947 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0291168Z [1699943405.013859] [99f90b6e6ff3:9947 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0291841Z [1699943405.016565] [99f90b6e6ff3:9948 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0292508Z [1699943405.016565] [99f90b6e6ff3:9948 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0293906Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0293946Z 2023-11-14T06:44:01.0294647Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0295347Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0295990Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0296618Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0297268Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0297974Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0298625Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0299339Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0299965Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0300742Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0301414Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0302060Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0302749Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0303452Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0304084Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0304727Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0305376Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0306023Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0306727Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0307416Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0308060Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0308688Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0309461Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0310152Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0310847Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0311541Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0312182Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0312812Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0313479Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0314882Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0314905Z 2023-11-14T06:44:01.0315594Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0316240Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0316865Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0317537Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0318179Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0318866Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0319573Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0320205Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0320830Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0321613Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0322262Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0322968Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0323653Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0324328Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0324976Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0325633Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0326296Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0326995Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0327682Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0328320Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0328943Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0329592Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0330248Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0331325Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0332074Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0332706Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0333329Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0333994Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0334639Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0335330Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0336030Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0336657Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0337306Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0337963Z STAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0339363Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:30:05 9947:9947 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0339385Z 2023-11-14T06:44:01.0340086Z STAGE:2023-11-14 06:30:05 9948:9948 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0340238Z PASSED [6.5024s] [100%] 2023-11-14T06:44:01.0340257Z 2023-11-14T06:44:01.0341518Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ad2f3b5b66e86a6.xml - 2023-11-14T06:44:01.0341788Z ============================== 1 passed in 6.54s =============================== 2023-11-14T06:44:01.0342799Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-960de975803b89cb.xml 2023-11-14T06:44:01.0343217Z ============================= test session starts ============================== 2023-11-14T06:44:01.0343867Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0344022Z cachedir: .pytest_cache 2023-11-14T06:44:01.0344880Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0345057Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0345228Z configfile: pytest.ini 2023-11-14T06:44:01.0346021Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0346199Z collecting ... collected 1 item 2023-11-14T06:44:01.0346978Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2023-11-14T06:44:01.0347004Z 2023-11-14T06:44:01.0348759Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.0349416Z STAGE:2023-11-14 06:30:16 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0350047Z STAGE:2023-11-14 06:30:16 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0350630Z [1699943416.630706] [99f90b6e6ff3:9970 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0351308Z [1699943416.631882] [99f90b6e6ff3:9971 :0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0351977Z [1699943416.752932] [99f90b6e6ff3:9971 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0352681Z [1699943416.752932] [99f90b6e6ff3:9971 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0353327Z [1699943416.753013] [99f90b6e6ff3:9970 :0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0353991Z [1699943416.753013] [99f90b6e6ff3:9970 :0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0355397Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0355418Z 2023-11-14T06:44:01.0356113Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0356832Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0357471Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0358127Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0358829Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0359458Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0360111Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0360751Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0361447Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0362091Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0362853Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0363500Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0364205Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0364895Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0365518Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0366159Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0366813Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0367475Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0368169Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0368859Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0369501Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0370125Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0371023Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0371689Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0372494Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0373221Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0373850Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0374476Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0375140Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0375787Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0376493Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0377183Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0377816Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0378460Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0379117Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0379763Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0380467Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0381158Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0381802Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0382428Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0383086Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0383750Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0384603Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0385310Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0385943Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0386568Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0387238Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0387880Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0388577Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0389283Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0389915Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0390557Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0391210Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0391854Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0392562Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0393247Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0393961Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0394628Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0395287Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0395948Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0396636Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0397322Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0397966Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0398594Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.0399249Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0399920Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.0400614Z STAGE:2023-11-14 06:30:17 9970:9970 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0401315Z STAGE:2023-11-14 06:30:17 9971:9971 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.0401465Z PASSED [6.5022s] [100%] 2023-11-14T06:44:01.0401486Z 2023-11-14T06:44:01.0402729Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-960de975803b89cb.xml - 2023-11-14T06:44:01.0403011Z ============================== 1 passed in 6.54s =============================== 2023-11-14T06:44:01.0404027Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c49b4ca7017676f.xml 2023-11-14T06:44:01.0404316Z ============================= test session starts ============================== 2023-11-14T06:44:01.0405072Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0405228Z cachedir: .pytest_cache 2023-11-14T06:44:01.0406091Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0406267Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0406422Z configfile: pytest.ini 2023-11-14T06:44:01.0407235Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0407404Z collecting ... collected 1 item 2023-11-14T06:44:01.0408127Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2023-11-14T06:44:01.0408166Z 2023-11-14T06:44:01.0409890Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.0410223Z SKIPPED [5.6113s] (Skipped due to small world size.) [100%] 2023-11-14T06:44:01.0410244Z 2023-11-14T06:44:01.0411734Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c49b4ca7017676f.xml - 2023-11-14T06:44:01.0412004Z ============================== 1 skipped in 5.65s ============================== 2023-11-14T06:44:01.0413037Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-909ac17954ca33b3.xml 2023-11-14T06:44:01.0413305Z ============================= test session starts ============================== 2023-11-14T06:44:01.0414072Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0414266Z cachedir: .pytest_cache 2023-11-14T06:44:01.0415119Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0415301Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0415471Z configfile: pytest.ini 2023-11-14T06:44:01.0416272Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0416460Z collecting ... collected 1 item 2023-11-14T06:44:01.0417208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_multigpu 2023-11-14T06:44:01.0417229Z 2023-11-14T06:44:01.0418972Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_multigpu <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.0419650Z [1699943439.510919] [99f90b6e6ff3:10006:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0420343Z [1699943439.510919] [99f90b6e6ff3:10006:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0421010Z [1699943439.503354] [99f90b6e6ff3:10005:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0421688Z [1699943439.503354] [99f90b6e6ff3:10005:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0421838Z PASSED [6.1058s] [100%] 2023-11-14T06:44:01.0421858Z 2023-11-14T06:44:01.0423118Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-909ac17954ca33b3.xml - 2023-11-14T06:44:01.0423387Z ============================== 1 passed in 6.14s =============================== 2023-11-14T06:44:01.0424477Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8bdad9fe71f8b49d.xml 2023-11-14T06:44:01.0424894Z ============================= test session starts ============================== 2023-11-14T06:44:01.0425546Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0425721Z cachedir: .pytest_cache 2023-11-14T06:44:01.0426560Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0426736Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0426908Z configfile: pytest.ini 2023-11-14T06:44:01.0427706Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0427875Z collecting ... collected 1 item 2023-11-14T06:44:01.0428665Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2023-11-14T06:44:01.0428686Z 2023-11-14T06:44:01.0430447Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.0431048Z [1699943450.893723] [99f90b6e6ff3:10028:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0431625Z [1699943450.914385] [99f90b6e6ff3:10029:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0432314Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0433178Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0434677Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0435435Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0436895Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0437500Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0438917Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0439637Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0441122Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0441873Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0443340Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0444077Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0445769Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0446732Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0448433Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0449187Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0450553Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0451621Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0453161Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0454065Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0455441Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0456183Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0457775Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0458649Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0460923Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0461471Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0462430Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0463552Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0464082Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0465136Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0465945Z [rank1]:[2023-11-14 06:30:51,405] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.0466605Z [1699943451.005633] [99f90b6e6ff3:10029:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0467303Z [1699943451.005633] [99f90b6e6ff3:10029:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0467972Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0468743Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0470233Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0471106Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0472581Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0473162Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0474581Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0475314Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0476776Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0477535Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0478993Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0479886Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0481593Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0482429Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0484145Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0484886Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0486271Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0487018Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0488559Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0489470Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0491078Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0491912Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0493369Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0494419Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0496679Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0497211Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0498198Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0499291Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0499838Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0500881Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0501667Z [rank0]:[2023-11-14 06:30:51,462] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.0502343Z [1699943451.002362] [99f90b6e6ff3:10028:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0503024Z [1699943451.002362] [99f90b6e6ff3:10028:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0503411Z ('RERUN', {'yellow': True}) [6.1102s] [100%] 2023-11-14T06:44:01.0505689Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [1699943456.721922] [99f90b6e6ff3:10044:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0506280Z [1699943456.742954] [99f90b6e6ff3:10045:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0506965Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0507736Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0509228Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0509965Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0511430Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0525303Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0526865Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0527613Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0529093Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0530052Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0531790Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0532562Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0534244Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0535081Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0536805Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0537549Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0538927Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0539669Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0541351Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0542305Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0543669Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0544432Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0545870Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0546737Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0548992Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0549531Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0550513Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0551613Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0552167Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0553214Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0554158Z [rank1]:[2023-11-14 06:30:57,226] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.0554841Z [1699943456.832684] [99f90b6e6ff3:10045:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0556742Z [1699943456.832684] [99f90b6e6ff3:10045:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0557474Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0558242Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0559733Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0560494Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0561955Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0562557Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0563972Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0564804Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0566363Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0567119Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0568597Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0569329Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0571301Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0572150Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0573858Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0574614Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0575970Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0576747Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0578280Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0579338Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0580720Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0581465Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0582936Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0583779Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0586066Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0586596Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0587552Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0588771Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0589321Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0590382Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0591173Z [rank0]:[2023-11-14 06:30:57,284] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.0591833Z [1699943456.830762] [99f90b6e6ff3:10044:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0592538Z [1699943456.830762] [99f90b6e6ff3:10044:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0592802Z ('RERUN', {'yellow': True}) [5.8133s] [100%] 2023-11-14T06:44:01.0595093Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [1699943462.577150] [99f90b6e6ff3:10060:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0595686Z [1699943462.583256] [99f90b6e6ff3:10061:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0596354Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0597143Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0598611Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0599371Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0600826Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0601530Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0602964Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0603676Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0605154Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0605903Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0607354Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0608110Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0609796Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0610644Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0612729Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0613532Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0614893Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0615633Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0617185Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0618083Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0619471Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0620222Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0621664Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0622526Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0624841Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0625535Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0626493Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0627604Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0628129Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0629182Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0629989Z [rank1]:[2023-11-14 06:31:03,091] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.0630649Z [1699943462.688229] [99f90b6e6ff3:10061:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0631356Z [1699943462.688229] [99f90b6e6ff3:10061:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0632028Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0632797Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0634280Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0635128Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0636620Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0637218Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0638631Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0639365Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0640822Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0641581Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0643038Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0643773Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0645468Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0646296Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0648012Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0648859Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0650237Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0651276Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0652833Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0653747Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0655119Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0655877Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0657321Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0658280Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0660586Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0661126Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0662102Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0663194Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0663724Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0664792Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0665577Z [rank0]:[2023-11-14 06:31:03,149] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.0666261Z [1699943462.686876] [99f90b6e6ff3:10060:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0666945Z [1699943462.686876] [99f90b6e6ff3:10060:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0667095Z FAILED [5.9135s] [100%] 2023-11-14T06:44:01.0667120Z 2023-11-14T06:44:01.0667382Z ==================================== RERUNS ==================================== 2023-11-14T06:44:01.0667790Z _____________ TestDistBackendWithSpawn.test_broadcast_object_list ______________ 2023-11-14T06:44:01.0667984Z Traceback (most recent call last): 2023-11-14T06:44:01.0668902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.0669065Z self._join_processes(fn) 2023-11-14T06:44:01.0670210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.0670404Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.0671397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.0671579Z raise RuntimeError(error) 2023-11-14T06:44:01.0671921Z RuntimeError: Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.0672118Z Traceback (most recent call last): 2023-11-14T06:44:01.0673030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0673192Z getattr(self, test_name)() 2023-11-14T06:44:01.0674113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0674247Z fn() 2023-11-14T06:44:01.0675113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0675286Z method(*args, **kwargs) 2023-11-14T06:44:01.0676174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0676338Z return func(*args, **kwargs) 2023-11-14T06:44:01.0677251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0677411Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.0678649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0678872Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.0680017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0680208Z dist.broadcast_object_list( 2023-11-14T06:44:01.0681013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0681177Z return func(*args, **kwargs) 2023-11-14T06:44:01.0682168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0682436Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0683263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0683427Z return func(*args, **kwargs) 2023-11-14T06:44:01.0684313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0684553Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0686280Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0686303Z 2023-11-14T06:44:01.0686653Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0687188Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0687208Z 2023-11-14T06:44:01.0687616Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0687636Z 2023-11-14T06:44:01.0687655Z 2023-11-14T06:44:01.0688058Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.0688460Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.0688838Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.0689269Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.0689671Z _____________ TestDistBackendWithSpawn.test_broadcast_object_list ______________ 2023-11-14T06:44:01.0689865Z Traceback (most recent call last): 2023-11-14T06:44:01.0691010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.0691186Z self._join_processes(fn) 2023-11-14T06:44:01.0692177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.0692370Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.0693371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.0693554Z raise RuntimeError(error) 2023-11-14T06:44:01.0693896Z RuntimeError: Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.0694095Z Traceback (most recent call last): 2023-11-14T06:44:01.0695011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0695171Z getattr(self, test_name)() 2023-11-14T06:44:01.0696088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0696224Z fn() 2023-11-14T06:44:01.0697082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0697258Z method(*args, **kwargs) 2023-11-14T06:44:01.0698270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0698497Z return func(*args, **kwargs) 2023-11-14T06:44:01.0699423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0699591Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.0700743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0700951Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.0702091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0702276Z dist.broadcast_object_list( 2023-11-14T06:44:01.0703076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0703247Z return func(*args, **kwargs) 2023-11-14T06:44:01.0704240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0704512Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0705336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0705501Z return func(*args, **kwargs) 2023-11-14T06:44:01.0706381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0706622Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0708339Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0708361Z 2023-11-14T06:44:01.0708715Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0709250Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0709394Z 2023-11-14T06:44:01.0709821Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0709841Z 2023-11-14T06:44:01.0709859Z 2023-11-14T06:44:01.0710267Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.0710660Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.0711059Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.0711329Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.0711710Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.0712119Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.0712365Z =================================== FAILURES =================================== 2023-11-14T06:44:01.0712764Z _____________ TestDistBackendWithSpawn.test_broadcast_object_list ______________ 2023-11-14T06:44:01.0712968Z Traceback (most recent call last): 2023-11-14T06:44:01.0713879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.0714040Z self._join_processes(fn) 2023-11-14T06:44:01.0715016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.0715209Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.0716223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.0716387Z raise RuntimeError(error) 2023-11-14T06:44:01.0716726Z RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.0717009Z Traceback (most recent call last): 2023-11-14T06:44:01.0717947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0718113Z getattr(self, test_name)() 2023-11-14T06:44:01.0719032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0719165Z fn() 2023-11-14T06:44:01.0720042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0720197Z method(*args, **kwargs) 2023-11-14T06:44:01.0721098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0721280Z return func(*args, **kwargs) 2023-11-14T06:44:01.0722184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0722345Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.0723491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0723707Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.0724918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0725088Z dist.broadcast_object_list( 2023-11-14T06:44:01.0725896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0726078Z return func(*args, **kwargs) 2023-11-14T06:44:01.0727051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0727326Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0728147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0728410Z return func(*args, **kwargs) 2023-11-14T06:44:01.0729299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0729539Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0731500Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0731525Z 2023-11-14T06:44:01.0731882Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0732415Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0732436Z 2023-11-14T06:44:01.0732850Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0732890Z 2023-11-14T06:44:01.0733137Z Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.0733319Z Traceback (most recent call last): 2023-11-14T06:44:01.0734255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0734417Z getattr(self, test_name)() 2023-11-14T06:44:01.0735321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0735473Z fn() 2023-11-14T06:44:01.0736331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0736485Z method(*args, **kwargs) 2023-11-14T06:44:01.0737515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0737698Z return func(*args, **kwargs) 2023-11-14T06:44:01.0738632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0738799Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.0739933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0740162Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.0741305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0741470Z dist.broadcast_object_list( 2023-11-14T06:44:01.0742290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0742454Z return func(*args, **kwargs) 2023-11-14T06:44:01.0743448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0743723Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0744524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0744708Z return func(*args, **kwargs) 2023-11-14T06:44:01.0745585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0745804Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0747537Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0747559Z 2023-11-14T06:44:01.0747893Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0748446Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0748578Z 2023-11-14T06:44:01.0749006Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0749027Z 2023-11-14T06:44:01.0749046Z 2023-11-14T06:44:01.0749436Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.0749847Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.0750226Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.0750511Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.0750892Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.0751282Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.0751689Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.0752076Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.0753326Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8bdad9fe71f8b49d.xml - 2023-11-14T06:44:01.0753618Z =========================== short test summary info ============================ 2023-11-14T06:44:01.0754815Z FAILED [5.9135s] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list - RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.0755010Z Traceback (most recent call last): 2023-11-14T06:44:01.0755926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0756088Z getattr(self, test_name)() 2023-11-14T06:44:01.0757097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0757241Z fn() 2023-11-14T06:44:01.0758112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0758292Z method(*args, **kwargs) 2023-11-14T06:44:01.0759193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0759375Z return func(*args, **kwargs) 2023-11-14T06:44:01.0760270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0760430Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.0761579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0761795Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.0762940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0763131Z dist.broadcast_object_list( 2023-11-14T06:44:01.0763935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0764117Z return func(*args, **kwargs) 2023-11-14T06:44:01.0765088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0765354Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0766175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0766341Z return func(*args, **kwargs) 2023-11-14T06:44:01.0767226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0767469Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0769184Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0769294Z 2023-11-14T06:44:01.0769655Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0770195Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0770215Z 2023-11-14T06:44:01.0770622Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0770643Z 2023-11-14T06:44:01.0771144Z Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.0771321Z Traceback (most recent call last): 2023-11-14T06:44:01.0772266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0772432Z getattr(self, test_name)() 2023-11-14T06:44:01.0773335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0773492Z fn() 2023-11-14T06:44:01.0774354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0774508Z method(*args, **kwargs) 2023-11-14T06:44:01.0775424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0775589Z return func(*args, **kwargs) 2023-11-14T06:44:01.0776505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0776667Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.0777960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0778220Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.0779367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0779531Z dist.broadcast_object_list( 2023-11-14T06:44:01.0780348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0780513Z return func(*args, **kwargs) 2023-11-14T06:44:01.0781506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0781772Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0782579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0782761Z return func(*args, **kwargs) 2023-11-14T06:44:01.0783646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0783872Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0785610Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0785630Z 2023-11-14T06:44:01.0785961Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0786509Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0786529Z 2023-11-14T06:44:01.0786936Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0787230Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2023-11-14T06:44:01.0787529Z ========================= 1 failed, 2 rerun in 17.88s ========================== 2023-11-14T06:44:01.0787872Z Got exit code 1, retrying (retries left=1) 2023-11-14T06:44:01.0788891Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-392198106c388806.xml 2023-11-14T06:44:01.0789179Z ============================= test session starts ============================== 2023-11-14T06:44:01.0789822Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.0789997Z cachedir: .pytest_cache 2023-11-14T06:44:01.0790840Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.0791017Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.0791189Z configfile: pytest.ini 2023-11-14T06:44:01.0792054Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.0792226Z collecting ... collected 1 item 2023-11-14T06:44:01.0793017Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2023-11-14T06:44:01.0793038Z 2023-11-14T06:44:01.0794796Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.0795407Z [1699943474.120974] [99f90b6e6ff3:10080:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0795984Z [1699943474.127123] [99f90b6e6ff3:10081:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0796742Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0797553Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0799038Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0799792Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0801246Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0801831Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0803272Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0803988Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0805470Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0806212Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0807684Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0808429Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0810112Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0811304Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0813022Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0813782Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0815147Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0815892Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0817451Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0818343Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0819727Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0820590Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0822088Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0822934Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0825249Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0825799Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0826762Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0827876Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0828408Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0829462Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0830269Z [rank1]:[2023-11-14 06:31:14,631] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.0830954Z [1699943474.232454] [99f90b6e6ff3:10081:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0831659Z [1699943474.232454] [99f90b6e6ff3:10081:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0832329Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0833238Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0834730Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0835465Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0836942Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0837532Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0838949Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0839689Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0841146Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0841906Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0843474Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0844234Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0845946Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0846768Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0848482Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0849226Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0850605Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0851699Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0853234Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0854142Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0855508Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0856272Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0857875Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0858723Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0861001Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0861537Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0862516Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0863616Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0864162Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0865207Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0865993Z [rank0]:[2023-11-14 06:31:14,690] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.0866670Z [1699943474.230925] [99f90b6e6ff3:10080:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0867472Z [1699943474.230925] [99f90b6e6ff3:10080:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0867784Z ('RERUN', {'yellow': True}) [6.2138s] [100%] 2023-11-14T06:44:01.0870043Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [1699943480.013760] [99f90b6e6ff3:10096:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0870626Z [1699943480.020256] [99f90b6e6ff3:10097:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0871311Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0872078Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0873571Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0874314Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0875772Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0876372Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0877786Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0878521Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0879973Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0880825Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0882295Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0883031Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0884739Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0885568Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0887290Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0888029Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0889389Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0890236Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0891964Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0892901Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0894262Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0895001Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0896470Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0897310Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0899593Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0900118Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0901087Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0902180Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0902704Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0903913Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0904699Z [rank1]:[2023-11-14 06:31:20,519] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.0905375Z [1699943480.125692] [99f90b6e6ff3:10097:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0906057Z [1699943480.125692] [99f90b6e6ff3:10097:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0906720Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0907512Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0908984Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0909747Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0911201Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0911786Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0913323Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0914060Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0915545Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0916287Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0917744Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0918499Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0920189Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0921043Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0922734Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0923472Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0924902Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0925652Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0927311Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0928202Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0929590Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0930331Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0932011Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0932894Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0935155Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0935704Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0936656Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0937881Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0938463Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0939506Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0940312Z [rank0]:[2023-11-14 06:31:20,575] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.0940971Z [1699943480.123701] [99f90b6e6ff3:10096:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0941651Z [1699943480.123701] [99f90b6e6ff3:10096:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0941936Z ('RERUN', {'yellow': True}) [5.8130s] [100%] 2023-11-14T06:44:01.0944189Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [1699943485.848271] [99f90b6e6ff3:10112:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0944797Z [1699943485.870577] [99f90b6e6ff3:10113:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.0945467Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0946234Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0947726Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0948461Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0950069Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0950651Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0952092Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0952802Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0954263Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0955030Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0956481Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0957234Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0958926Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0959838Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0961571Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0962319Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0963696Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0964438Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0965971Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.0966883Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.0968248Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0969009Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0970446Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.0971548Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.0973820Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.0974497Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0975471Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.0976560Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.0977102Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.0978158Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.0978949Z [rank1]:[2023-11-14 06:31:26,358] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.0979625Z [1699943485.961820] [99f90b6e6ff3:10113:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.0980309Z [1699943485.961820] [99f90b6e6ff3:10113:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.0980992Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.0981760Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.0983344Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.0984131Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.0985586Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.0986188Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.0987604Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.0988322Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.0989797Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.0990544Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.0992013Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.0992745Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] ret = func(*args, **kwargs) 2023-11-14T06:44:01.0994449Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.0995281Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] return self._test_broadcast_object_list() 2023-11-14T06:44:01.0997086Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.0997840Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] dist.broadcast_object_list( 2023-11-14T06:44:01.0999196Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.0999962Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1001499Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.1002395Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.1003774Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1004514Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1006054Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.1006915Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.1009209Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.1009739Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1010903Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1012059Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.1012587Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1013649Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1014438Z [rank0]:[2023-11-14 06:31:26,415] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.1015095Z [1699943485.960877] [99f90b6e6ff3:10112:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1015798Z [1699943485.960877] [99f90b6e6ff3:10112:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1015949Z FAILED [5.9135s] [100%] 2023-11-14T06:44:01.1015971Z 2023-11-14T06:44:01.1016233Z ==================================== RERUNS ==================================== 2023-11-14T06:44:01.1016645Z _____________ TestDistBackendWithSpawn.test_broadcast_object_list ______________ 2023-11-14T06:44:01.1016823Z Traceback (most recent call last): 2023-11-14T06:44:01.1017752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.1018065Z self._join_processes(fn) 2023-11-14T06:44:01.1019041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.1019259Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.1020256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.1020422Z raise RuntimeError(error) 2023-11-14T06:44:01.1020781Z RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.1020956Z Traceback (most recent call last): 2023-11-14T06:44:01.1021892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1022058Z getattr(self, test_name)() 2023-11-14T06:44:01.1022966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1023117Z fn() 2023-11-14T06:44:01.1023976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1024131Z method(*args, **kwargs) 2023-11-14T06:44:01.1025089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.1025256Z return func(*args, **kwargs) 2023-11-14T06:44:01.1026173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.1026334Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.1027610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.1027864Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.1029011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.1029175Z dist.broadcast_object_list( 2023-11-14T06:44:01.1029999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1030162Z return func(*args, **kwargs) 2023-11-14T06:44:01.1031151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.1031421Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.1032230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1032414Z return func(*args, **kwargs) 2023-11-14T06:44:01.1033299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.1033527Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.1035263Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.1035285Z 2023-11-14T06:44:01.1035615Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1036164Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.1036184Z 2023-11-14T06:44:01.1036593Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1036620Z 2023-11-14T06:44:01.1036869Z Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.1037066Z Traceback (most recent call last): 2023-11-14T06:44:01.1038095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1038276Z getattr(self, test_name)() 2023-11-14T06:44:01.1039178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1039311Z fn() 2023-11-14T06:44:01.1040194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1040350Z method(*args, **kwargs) 2023-11-14T06:44:01.1041242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.1041427Z return func(*args, **kwargs) 2023-11-14T06:44:01.1042328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.1042490Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.1043645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.1043855Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.1045017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.1045184Z dist.broadcast_object_list( 2023-11-14T06:44:01.1045986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1046171Z return func(*args, **kwargs) 2023-11-14T06:44:01.1047230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.1047511Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.1048336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1048506Z return func(*args, **kwargs) 2023-11-14T06:44:01.1049411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.1049633Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.1051585Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.1051610Z 2023-11-14T06:44:01.1051967Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1052512Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.1052532Z 2023-11-14T06:44:01.1052959Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1052985Z 2023-11-14T06:44:01.1053004Z 2023-11-14T06:44:01.1053389Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1053783Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1054180Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.1054449Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.1054852Z _____________ TestDistBackendWithSpawn.test_broadcast_object_list ______________ 2023-11-14T06:44:01.1055047Z Traceback (most recent call last): 2023-11-14T06:44:01.1055956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.1056146Z self._join_processes(fn) 2023-11-14T06:44:01.1057110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.1057446Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.1058465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.1058631Z raise RuntimeError(error) 2023-11-14T06:44:01.1058973Z RuntimeError: Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.1059169Z Traceback (most recent call last): 2023-11-14T06:44:01.1060078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1060259Z getattr(self, test_name)() 2023-11-14T06:44:01.1061154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1061294Z fn() 2023-11-14T06:44:01.1062177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1062339Z method(*args, **kwargs) 2023-11-14T06:44:01.1063231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.1063416Z return func(*args, **kwargs) 2023-11-14T06:44:01.1064317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.1064477Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.1065625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.1065838Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.1067103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.1067283Z dist.broadcast_object_list( 2023-11-14T06:44:01.1068102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1068288Z return func(*args, **kwargs) 2023-11-14T06:44:01.1069262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.1069547Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.1070351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1070515Z return func(*args, **kwargs) 2023-11-14T06:44:01.1071411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.1071639Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.1073355Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.1073383Z 2023-11-14T06:44:01.1073732Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1074262Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.1074282Z 2023-11-14T06:44:01.1074709Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1074729Z 2023-11-14T06:44:01.1074748Z 2023-11-14T06:44:01.1075130Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1075523Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1075926Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.1076195Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.1076594Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1077094Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1077334Z =================================== FAILURES =================================== 2023-11-14T06:44:01.1077757Z _____________ TestDistBackendWithSpawn.test_broadcast_object_list ______________ 2023-11-14T06:44:01.1077932Z Traceback (most recent call last): 2023-11-14T06:44:01.1078844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.1079026Z self._join_processes(fn) 2023-11-14T06:44:01.1079984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.1080184Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.1081198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.1081371Z raise RuntimeError(error) 2023-11-14T06:44:01.1081730Z RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.1081906Z Traceback (most recent call last): 2023-11-14T06:44:01.1082822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1083003Z getattr(self, test_name)() 2023-11-14T06:44:01.1083897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1084028Z fn() 2023-11-14T06:44:01.1084905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1085176Z method(*args, **kwargs) 2023-11-14T06:44:01.1086116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.1086288Z return func(*args, **kwargs) 2023-11-14T06:44:01.1087188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.1087365Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.1088491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.1088701Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.1089858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.1090021Z dist.broadcast_object_list( 2023-11-14T06:44:01.1091083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1091260Z return func(*args, **kwargs) 2023-11-14T06:44:01.1092295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.1092583Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.1093381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1093546Z return func(*args, **kwargs) 2023-11-14T06:44:01.1094444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.1094663Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.1096401Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.1096424Z 2023-11-14T06:44:01.1096752Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1097431Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.1097451Z 2023-11-14T06:44:01.1097882Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1097902Z 2023-11-14T06:44:01.1098149Z Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.1098344Z Traceback (most recent call last): 2023-11-14T06:44:01.1099258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1099419Z getattr(self, test_name)() 2023-11-14T06:44:01.1100337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1100476Z fn() 2023-11-14T06:44:01.1101334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1101516Z method(*args, **kwargs) 2023-11-14T06:44:01.1102422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.1102586Z return func(*args, **kwargs) 2023-11-14T06:44:01.1103504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.1103670Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.1104818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.1105026Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.1106275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.1106476Z dist.broadcast_object_list( 2023-11-14T06:44:01.1107292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1107458Z return func(*args, **kwargs) 2023-11-14T06:44:01.1108446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.1108713Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.1109535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1109700Z return func(*args, **kwargs) 2023-11-14T06:44:01.1110583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.1110830Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.1112541Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.1112568Z 2023-11-14T06:44:01.1112917Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1113448Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.1113468Z 2023-11-14T06:44:01.1113877Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1113898Z 2023-11-14T06:44:01.1113917Z 2023-11-14T06:44:01.1114326Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1114721Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1115103Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.1115441Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.1115820Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1116335Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1116721Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1117108Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1118360Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-392198106c388806.xml - 2023-11-14T06:44:01.1118634Z =========================== short test summary info ============================ 2023-11-14T06:44:01.1119854Z FAILED [5.9135s] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list - RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.1120033Z Traceback (most recent call last): 2023-11-14T06:44:01.1120949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1121135Z getattr(self, test_name)() 2023-11-14T06:44:01.1122035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1122168Z fn() 2023-11-14T06:44:01.1123043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1123198Z method(*args, **kwargs) 2023-11-14T06:44:01.1124085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.1124265Z return func(*args, **kwargs) 2023-11-14T06:44:01.1125321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.1125513Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.1126651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.1126870Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.1128032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.1128196Z dist.broadcast_object_list( 2023-11-14T06:44:01.1128995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1129177Z return func(*args, **kwargs) 2023-11-14T06:44:01.1130155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.1130443Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.1131473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1131651Z return func(*args, **kwargs) 2023-11-14T06:44:01.1132557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.1132776Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.1134489Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 0][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.1134531Z 2023-11-14T06:44:01.1134861Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1135389Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.1135416Z 2023-11-14T06:44:01.1135847Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1135867Z 2023-11-14T06:44:01.1136256Z Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.1136431Z Traceback (most recent call last): 2023-11-14T06:44:01.1137373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1137535Z getattr(self, test_name)() 2023-11-14T06:44:01.1138451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1138582Z fn() 2023-11-14T06:44:01.1139440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1139615Z method(*args, **kwargs) 2023-11-14T06:44:01.1140518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 143, in wrapper 2023-11-14T06:44:01.1140684Z return func(*args, **kwargs) 2023-11-14T06:44:01.1141599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 274, in wrapper 2023-11-14T06:44:01.1141765Z ret = func(*args, **kwargs) 2023-11-14T06:44:01.1142915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7717, in test_broadcast_object_list 2023-11-14T06:44:01.1143128Z return self._test_broadcast_object_list() 2023-11-14T06:44:01.1144269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 7683, in _test_broadcast_object_list 2023-11-14T06:44:01.1144454Z dist.broadcast_object_list( 2023-11-14T06:44:01.1145254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1145532Z return func(*args, **kwargs) 2023-11-14T06:44:01.1146551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2650, in broadcast_object_list 2023-11-14T06:44:01.1146824Z broadcast(object_sizes_tensor, src=src, group=group) 2023-11-14T06:44:01.1147649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper 2023-11-14T06:44:01.1147813Z return func(*args, **kwargs) 2023-11-14T06:44:01.1148692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1947, in broadcast 2023-11-14T06:44:01.1148931Z work = default_pg.broadcast([tensor], opts) 2023-11-14T06:44:01.1150648Z RuntimeError: [/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupUCC.cpp:488] [Rank 1][ProcessGroupUCC-0][READY]failed to init cuda collective, error code -1: Operation is not supported, system error code 2 2023-11-14T06:44:01.1150669Z 2023-11-14T06:44:01.1151004Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1151553Z python test/distributed/test_distributed_spawn.py -k test_broadcast_object_list 2023-11-14T06:44:01.1151579Z 2023-11-14T06:44:01.1151987Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1152295Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2023-11-14T06:44:01.1152574Z ========================= 1 failed, 2 rerun in 17.99s ========================== 2023-11-14T06:44:01.1155544Z Test exited with non-zero exitcode 1. Command to reproduce: BACKEND=ucc WORLD_SIZE=2 /opt/conda/envs/py_3.10/bin/python distributed/test_distributed_spawn.py --import-disabled-tests --import-slow-tests --use-pytest --shard-id=0 --num-shards=2 -v -vv -rfEX -p no:xdist -x --reruns=2 --pytest-single-test distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2023-11-14T06:44:01.1156587Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fad1e8098225ecaa.xml 2023-11-14T06:44:01.1156858Z ============================= test session starts ============================== 2023-11-14T06:44:01.1157644Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1157799Z cachedir: .pytest_cache 2023-11-14T06:44:01.1158641Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1158837Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1158992Z configfile: pytest.ini 2023-11-14T06:44:01.1159787Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1159974Z collecting ... collected 1 item 2023-11-14T06:44:01.1160727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2023-11-14T06:44:01.1160749Z 2023-11-14T06:44:01.1162937Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Coalescing manager currently tests with NCCL only; internal test flaky) [100%] 2023-11-14T06:44:01.1162965Z 2023-11-14T06:44:01.1164205Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fad1e8098225ecaa.xml - 2023-11-14T06:44:01.1164487Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.1165497Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-957bc4d267df0b1c.xml 2023-11-14T06:44:01.1165763Z ============================= test session starts ============================== 2023-11-14T06:44:01.1166509Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1166680Z cachedir: .pytest_cache 2023-11-14T06:44:01.1167534Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1167732Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1167886Z configfile: pytest.ini 2023-11-14T06:44:01.1168701Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1168873Z collecting ... collected 1 item 2023-11-14T06:44:01.1169656Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2023-11-14T06:44:01.1169678Z 2023-11-14T06:44:01.1172138Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Coalescing manager currently tests with NCCL only; internal test flaky) [100%] 2023-11-14T06:44:01.1172168Z 2023-11-14T06:44:01.1173434Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-957bc4d267df0b1c.xml - 2023-11-14T06:44:01.1173718Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.1174730Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b180c6f34341c07e.xml 2023-11-14T06:44:01.1174996Z ============================= test session starts ============================== 2023-11-14T06:44:01.1175654Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1175810Z cachedir: .pytest_cache 2023-11-14T06:44:01.1176679Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1176857Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1177189Z configfile: pytest.ini 2023-11-14T06:44:01.1178011Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1178179Z collecting ... collected 1 item 2023-11-14T06:44:01.1179170Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2023-11-14T06:44:01.1179191Z 2023-11-14T06:44:01.1181191Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1181855Z [1699943508.173658] [99f90b6e6ff3:10139:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1182559Z [1699943508.173658] [99f90b6e6ff3:10139:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1183208Z [1699943508.163970] [99f90b6e6ff3:10138:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1183879Z [1699943508.163970] [99f90b6e6ff3:10138:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1184053Z PASSED [6.2099s] [100%] 2023-11-14T06:44:01.1184073Z 2023-11-14T06:44:01.1185318Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b180c6f34341c07e.xml - 2023-11-14T06:44:01.1185600Z ============================== 1 passed in 6.25s =============================== 2023-11-14T06:44:01.1186718Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b2c1fbe2a5ee19a.xml 2023-11-14T06:44:01.1187006Z ============================= test session starts ============================== 2023-11-14T06:44:01.1187683Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1187840Z cachedir: .pytest_cache 2023-11-14T06:44:01.1188701Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1188880Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1189032Z configfile: pytest.ini 2023-11-14T06:44:01.1189848Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1190017Z collecting ... collected 1 item 2023-11-14T06:44:01.1190982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2023-11-14T06:44:01.1191004Z 2023-11-14T06:44:01.1192972Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1193636Z [1699943519.575240] [99f90b6e6ff3:10167:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1194340Z [1699943519.575240] [99f90b6e6ff3:10167:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1194985Z [1699943519.582859] [99f90b6e6ff3:10168:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1195662Z [1699943519.582859] [99f90b6e6ff3:10168:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1195834Z PASSED [6.5099s] [100%] 2023-11-14T06:44:01.1195854Z 2023-11-14T06:44:01.1197108Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b2c1fbe2a5ee19a.xml - 2023-11-14T06:44:01.1197494Z ============================== 1 passed in 6.55s =============================== 2023-11-14T06:44:01.1198511Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1bac0f240ea01924.xml 2023-11-14T06:44:01.1198779Z ============================= test session starts ============================== 2023-11-14T06:44:01.1199441Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1199597Z cachedir: .pytest_cache 2023-11-14T06:44:01.1200462Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1200641Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1200795Z configfile: pytest.ini 2023-11-14T06:44:01.1201611Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1201787Z collecting ... collected 1 item 2023-11-14T06:44:01.1202682Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2023-11-14T06:44:01.1202704Z 2023-11-14T06:44:01.1204602Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1206975Z SKIPPED [0.2824s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/106361 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [100%] 2023-11-14T06:44:01.1207005Z 2023-11-14T06:44:01.1208295Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1bac0f240ea01924.xml - 2023-11-14T06:44:01.1208562Z ============================== 1 skipped in 0.32s ============================== 2023-11-14T06:44:01.1209595Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fea7b0c4be027a8e.xml 2023-11-14T06:44:01.1209864Z ============================= test session starts ============================== 2023-11-14T06:44:01.1210507Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1210849Z cachedir: .pytest_cache 2023-11-14T06:44:01.1211719Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1211896Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1212075Z configfile: pytest.ini 2023-11-14T06:44:01.1212876Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1213064Z collecting ... collected 1 item 2023-11-14T06:44:01.1213827Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2023-11-14T06:44:01.1213848Z 2023-11-14T06:44:01.1215592Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1216281Z [1699943537.025440] [99f90b6e6ff3:10198:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1216961Z [1699943537.025440] [99f90b6e6ff3:10198:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1217776Z [1699943537.018993] [99f90b6e6ff3:10197:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1218455Z [1699943537.018993] [99f90b6e6ff3:10197:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1218606Z PASSED [6.2120s] [100%] 2023-11-14T06:44:01.1218626Z 2023-11-14T06:44:01.1219899Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fea7b0c4be027a8e.xml - 2023-11-14T06:44:01.1220162Z ============================== 1 passed in 6.25s =============================== 2023-11-14T06:44:01.1221198Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e6975418dcfc618.xml 2023-11-14T06:44:01.1221467Z ============================= test session starts ============================== 2023-11-14T06:44:01.1222117Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1222293Z cachedir: .pytest_cache 2023-11-14T06:44:01.1223133Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1223311Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1223483Z configfile: pytest.ini 2023-11-14T06:44:01.1224282Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1224495Z collecting ... collected 1 item 2023-11-14T06:44:01.1225329Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2023-11-14T06:44:01.1225467Z 2023-11-14T06:44:01.1227295Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1227979Z [1699943548.536773] [99f90b6e6ff3:10225:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1228660Z [1699943548.536773] [99f90b6e6ff3:10225:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1229331Z [1699943548.529345] [99f90b6e6ff3:10224:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1230010Z [1699943548.529345] [99f90b6e6ff3:10224:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1230160Z PASSED [6.2122s] [100%] 2023-11-14T06:44:01.1230187Z 2023-11-14T06:44:01.1231449Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e6975418dcfc618.xml - 2023-11-14T06:44:01.1231722Z ============================== 1 passed in 6.25s =============================== 2023-11-14T06:44:01.1232738Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-223cd7ec5fa2cfe5.xml 2023-11-14T06:44:01.1233026Z ============================= test session starts ============================== 2023-11-14T06:44:01.1233665Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1233839Z cachedir: .pytest_cache 2023-11-14T06:44:01.1234678Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1234853Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1235033Z configfile: pytest.ini 2023-11-14T06:44:01.1235827Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1236094Z collecting ... collected 1 item 2023-11-14T06:44:01.1236898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2023-11-14T06:44:01.1236919Z 2023-11-14T06:44:01.1238703Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1239378Z [1699943560.132962] [99f90b6e6ff3:10251:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1240065Z [1699943560.132962] [99f90b6e6ff3:10251:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1240715Z [1699943560.135192] [99f90b6e6ff3:10252:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1241420Z [1699943560.135192] [99f90b6e6ff3:10252:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1241570Z PASSED [6.3131s] [100%] 2023-11-14T06:44:01.1241590Z 2023-11-14T06:44:01.1242860Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-223cd7ec5fa2cfe5.xml - 2023-11-14T06:44:01.1243125Z ============================== 1 passed in 6.35s =============================== 2023-11-14T06:44:01.1244136Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a882cd33429420c4.xml 2023-11-14T06:44:01.1244421Z ============================= test session starts ============================== 2023-11-14T06:44:01.1245149Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1245335Z cachedir: .pytest_cache 2023-11-14T06:44:01.1246189Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1246364Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1246540Z configfile: pytest.ini 2023-11-14T06:44:01.1247334Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1247503Z collecting ... collected 1 item 2023-11-14T06:44:01.1248476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2023-11-14T06:44:01.1248497Z 2023-11-14T06:44:01.1250434Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1251442Z [1699943571.709469] [99f90b6e6ff3:10279:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1252139Z [1699943571.709469] [99f90b6e6ff3:10279:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1252784Z [1699943571.707741] [99f90b6e6ff3:10278:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1253482Z [1699943571.707741] [99f90b6e6ff3:10278:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1253633Z PASSED [6.1008s] [100%] 2023-11-14T06:44:01.1253653Z 2023-11-14T06:44:01.1254927Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a882cd33429420c4.xml - 2023-11-14T06:44:01.1255192Z ============================== 1 passed in 6.14s =============================== 2023-11-14T06:44:01.1256354Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c13b3ef4915302e1.xml 2023-11-14T06:44:01.1256642Z ============================= test session starts ============================== 2023-11-14T06:44:01.1257282Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1257456Z cachedir: .pytest_cache 2023-11-14T06:44:01.1258295Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1258471Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1258644Z configfile: pytest.ini 2023-11-14T06:44:01.1259445Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1259616Z collecting ... collected 1 item 2023-11-14T06:44:01.1260459Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2023-11-14T06:44:01.1260486Z 2023-11-14T06:44:01.1262299Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1262983Z [1699943583.221143] [99f90b6e6ff3:10302:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1263666Z [1699943583.221143] [99f90b6e6ff3:10302:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1264421Z [1699943583.216274] [99f90b6e6ff3:10301:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1265134Z [1699943583.216274] [99f90b6e6ff3:10301:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1265292Z PASSED [6.2024s] [100%] 2023-11-14T06:44:01.1265312Z 2023-11-14T06:44:01.1266576Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c13b3ef4915302e1.xml - 2023-11-14T06:44:01.1266838Z ============================== 1 passed in 6.24s =============================== 2023-11-14T06:44:01.1267845Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1ba40569b19dbdd.xml 2023-11-14T06:44:01.1268132Z ============================= test session starts ============================== 2023-11-14T06:44:01.1268774Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1268954Z cachedir: .pytest_cache 2023-11-14T06:44:01.1269794Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1269977Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1270152Z configfile: pytest.ini 2023-11-14T06:44:01.1270951Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1271119Z collecting ... collected 1 item 2023-11-14T06:44:01.1271937Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2023-11-14T06:44:01.1271957Z 2023-11-14T06:44:01.1273736Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1274420Z [1699943594.790529] [99f90b6e6ff3:10325:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1275101Z [1699943594.790529] [99f90b6e6ff3:10325:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1275868Z [1699943594.785380] [99f90b6e6ff3:10324:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1276565Z [1699943594.785380] [99f90b6e6ff3:10324:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1277546Z /opt/conda/envs/py_3.10/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up 2023-11-14T06:44:01.1277795Z _warnings.warn(warn_message, ResourceWarning) 2023-11-14T06:44:01.1278765Z /opt/conda/envs/py_3.10/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up 2023-11-14T06:44:01.1279001Z _warnings.warn(warn_message, ResourceWarning) 2023-11-14T06:44:01.1279171Z PASSED [6.3137s] [100%] 2023-11-14T06:44:01.1279191Z 2023-11-14T06:44:01.1280434Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1ba40569b19dbdd.xml - 2023-11-14T06:44:01.1280708Z ============================== 1 passed in 6.35s =============================== 2023-11-14T06:44:01.1281731Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71585e8e03013eee.xml 2023-11-14T06:44:01.1281998Z ============================= test session starts ============================== 2023-11-14T06:44:01.1282664Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1282824Z cachedir: .pytest_cache 2023-11-14T06:44:01.1283753Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1283967Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1284120Z configfile: pytest.ini 2023-11-14T06:44:01.1284934Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1285123Z collecting ... collected 1 item 2023-11-14T06:44:01.1286005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2023-11-14T06:44:01.1286027Z 2023-11-14T06:44:01.1287914Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1288578Z [1699943606.300342] [99f90b6e6ff3:10352:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1289259Z [1699943606.300342] [99f90b6e6ff3:10352:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1289925Z [1699943606.293789] [99f90b6e6ff3:10351:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1290592Z [1699943606.293789] [99f90b6e6ff3:10351:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1290985Z PASSED [6.5147s] [100%] 2023-11-14T06:44:01.1291008Z 2023-11-14T06:44:01.1292266Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71585e8e03013eee.xml - 2023-11-14T06:44:01.1292530Z ============================== 1 passed in 6.55s =============================== 2023-11-14T06:44:01.1293568Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fbf4f05b0440cddd.xml 2023-11-14T06:44:01.1293839Z ============================= test session starts ============================== 2023-11-14T06:44:01.1294668Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1294824Z cachedir: .pytest_cache 2023-11-14T06:44:01.1295665Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1295864Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1296016Z configfile: pytest.ini 2023-11-14T06:44:01.1296809Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1296996Z collecting ... collected 1 item 2023-11-14T06:44:01.1297814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2023-11-14T06:44:01.1297836Z 2023-11-14T06:44:01.1299660Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1300324Z [1699943618.115861] [99f90b6e6ff3:10379:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1301008Z [1699943618.115861] [99f90b6e6ff3:10379:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1301676Z [1699943618.108668] [99f90b6e6ff3:10378:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1302350Z [1699943618.108668] [99f90b6e6ff3:10378:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1302519Z PASSED [6.9124s] [100%] 2023-11-14T06:44:01.1302538Z 2023-11-14T06:44:01.1303896Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fbf4f05b0440cddd.xml - 2023-11-14T06:44:01.1304185Z ============================== 1 passed in 6.95s =============================== 2023-11-14T06:44:01.1305222Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48877715bcf9f5ed.xml 2023-11-14T06:44:01.1305491Z ============================= test session starts ============================== 2023-11-14T06:44:01.1306153Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1306307Z cachedir: .pytest_cache 2023-11-14T06:44:01.1307148Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1307345Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1307497Z configfile: pytest.ini 2023-11-14T06:44:01.1308310Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1308501Z collecting ... collected 1 item 2023-11-14T06:44:01.1309219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2023-11-14T06:44:01.1309240Z 2023-11-14T06:44:01.1310962Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1311619Z [1699943630.387796] [99f90b6e6ff3:10405:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1312301Z [1699943630.387796] [99f90b6e6ff3:10405:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1312975Z [1699943630.397302] [99f90b6e6ff3:10406:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1313655Z [1699943630.397302] [99f90b6e6ff3:10406:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1313922Z PASSED [6.4155s] [100%] 2023-11-14T06:44:01.1313943Z 2023-11-14T06:44:01.1315188Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48877715bcf9f5ed.xml - 2023-11-14T06:44:01.1315452Z ============================== 1 passed in 6.45s =============================== 2023-11-14T06:44:01.1316484Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c34d29cb2e8604d7.xml 2023-11-14T06:44:01.1316751Z ============================= test session starts ============================== 2023-11-14T06:44:01.1317419Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1317577Z cachedir: .pytest_cache 2023-11-14T06:44:01.1318420Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1318619Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1318772Z configfile: pytest.ini 2023-11-14T06:44:01.1319570Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1319759Z collecting ... collected 1 item 2023-11-14T06:44:01.1320556Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2023-11-14T06:44:01.1320577Z 2023-11-14T06:44:01.1322465Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1323138Z [1699943642.146646] [99f90b6e6ff3:10428:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1323827Z [1699943642.146646] [99f90b6e6ff3:10428:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1324546Z [1699943642.155973] [99f90b6e6ff3:10429:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1325228Z [1699943642.155973] [99f90b6e6ff3:10429:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1326235Z /opt/conda/envs/py_3.10/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up 2023-11-14T06:44:01.1326465Z _warnings.warn(warn_message, ResourceWarning) 2023-11-14T06:44:01.1327446Z /opt/conda/envs/py_3.10/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up 2023-11-14T06:44:01.1327694Z _warnings.warn(warn_message, ResourceWarning) 2023-11-14T06:44:01.1327848Z PASSED [6.3014s] [100%] 2023-11-14T06:44:01.1327868Z 2023-11-14T06:44:01.1329117Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c34d29cb2e8604d7.xml - 2023-11-14T06:44:01.1329403Z ============================== 1 passed in 6.34s =============================== 2023-11-14T06:44:01.1330408Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-feb86165052942eb.xml 2023-11-14T06:44:01.1330912Z ============================= test session starts ============================== 2023-11-14T06:44:01.1331584Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1331742Z cachedir: .pytest_cache 2023-11-14T06:44:01.1332610Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1332788Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1333121Z configfile: pytest.ini 2023-11-14T06:44:01.1333944Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1334113Z collecting ... collected 1 item 2023-11-14T06:44:01.1334892Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2023-11-14T06:44:01.1334913Z 2023-11-14T06:44:01.1336654Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1337244Z [1699943653.648107] [99f90b6e6ff3:10455:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.1337838Z [1699943653.648457] [99f90b6e6ff3:10456:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.1338499Z [1699943653.760601] [99f90b6e6ff3:10455:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1339198Z [1699943653.760601] [99f90b6e6ff3:10455:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1339845Z [1699943653.760541] [99f90b6e6ff3:10456:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1340522Z [1699943653.760541] [99f90b6e6ff3:10456:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1340691Z PASSED [6.2110s] [100%] 2023-11-14T06:44:01.1340710Z 2023-11-14T06:44:01.1342058Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-feb86165052942eb.xml - 2023-11-14T06:44:01.1342359Z ============================== 1 passed in 6.25s =============================== 2023-11-14T06:44:01.1343379Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d71b8154e337ced9.xml 2023-11-14T06:44:01.1343645Z ============================= test session starts ============================== 2023-11-14T06:44:01.1344307Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1344462Z cachedir: .pytest_cache 2023-11-14T06:44:01.1345304Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1345503Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1345656Z configfile: pytest.ini 2023-11-14T06:44:01.1346477Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1346648Z collecting ... collected 1 item 2023-11-14T06:44:01.1347406Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2023-11-14T06:44:01.1347432Z 2023-11-14T06:44:01.1349200Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1349858Z [1699943665.169532] [99f90b6e6ff3:10508:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1350562Z [1699943665.169532] [99f90b6e6ff3:10508:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1351215Z [1699943665.173751] [99f90b6e6ff3:10509:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1351894Z [1699943665.173751] [99f90b6e6ff3:10509:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1352170Z PASSED [6.3107s] [100%] 2023-11-14T06:44:01.1352192Z 2023-11-14T06:44:01.1353444Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d71b8154e337ced9.xml - 2023-11-14T06:44:01.1353726Z ============================== 1 passed in 6.35s =============================== 2023-11-14T06:44:01.1354737Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f3059cefd2e68e4.xml 2023-11-14T06:44:01.1355002Z ============================= test session starts ============================== 2023-11-14T06:44:01.1355808Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1355972Z cachedir: .pytest_cache 2023-11-14T06:44:01.1356820Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1357023Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1357178Z configfile: pytest.ini 2023-11-14T06:44:01.1358248Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1358425Z collecting ... collected 1 item 2023-11-14T06:44:01.1359266Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2023-11-14T06:44:01.1360372Z 2023-11-14T06:44:01.1362262Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1363038Z [1699943676.880021] [99f90b6e6ff3:10536:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1363765Z [1699943676.880021] [99f90b6e6ff3:10536:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1364621Z [1699943687.316603] [99f90b6e6ff3:10536:0] tag_match.c:62 UCX WARN unexpected tag-receive descriptor 0x9ad8780 was not matched 2023-11-14T06:44:01.1365269Z [1699943676.879798] [99f90b6e6ff3:10535:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1365961Z [1699943676.879798] [99f90b6e6ff3:10535:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1366993Z [1699943687.292887] [99f90b6e6ff3:10535:1] ucc_schedule.h:192 UCC WARN timeout 10 sec. has expired on Barrier; CL_BASIC {TL_UCP}, team_id 1 rank 0, ctx_rank 0, seq_num 3, req 0x8a6fe40 2023-11-14T06:44:01.1367919Z [1699943687.326624] [99f90b6e6ff3:10535:0] mpool.c:54 UCX WARN object 0x8b7a340 {{cb|rcv_tag} recv length 0 host memory} was not returned to mpool ucp_requests 2023-11-14T06:44:01.1368082Z PASSED [16.2261s] [100%] 2023-11-14T06:44:01.1368103Z 2023-11-14T06:44:01.1369353Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f3059cefd2e68e4.xml - 2023-11-14T06:44:01.1369638Z ============================== 1 passed in 16.26s ============================== 2023-11-14T06:44:01.1370654Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34fd237fdc95e7f2.xml 2023-11-14T06:44:01.1371175Z ============================= test session starts ============================== 2023-11-14T06:44:01.1371824Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1371987Z cachedir: .pytest_cache 2023-11-14T06:44:01.1372849Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1373172Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1373325Z configfile: pytest.ini 2023-11-14T06:44:01.1374151Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1374321Z collecting ... collected 1 item 2023-11-14T06:44:01.1375292Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2023-11-14T06:44:01.1375314Z 2023-11-14T06:44:01.1377258Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1377916Z [1699943698.319278] [99f90b6e6ff3:10564:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1378622Z [1699943698.319278] [99f90b6e6ff3:10564:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1379262Z [1699943698.327198] [99f90b6e6ff3:10565:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1379949Z [1699943698.327198] [99f90b6e6ff3:10565:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1380100Z PASSED [6.1996s] [100%] 2023-11-14T06:44:01.1380120Z 2023-11-14T06:44:01.1381372Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34fd237fdc95e7f2.xml - 2023-11-14T06:44:01.1381763Z ============================== 1 passed in 6.24s =============================== 2023-11-14T06:44:01.1382916Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15ac746e6903e41c.xml 2023-11-14T06:44:01.1383212Z ============================= test session starts ============================== 2023-11-14T06:44:01.1383857Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1384015Z cachedir: .pytest_cache 2023-11-14T06:44:01.1384876Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1385051Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1385207Z configfile: pytest.ini 2023-11-14T06:44:01.1386019Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1386189Z collecting ... collected 1 item 2023-11-14T06:44:01.1387228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2023-11-14T06:44:01.1387255Z 2023-11-14T06:44:01.1389265Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1390659Z [rank0]:[2023-11-14 06:35:09,770] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:44:01.1392121Z [rank1]:[2023-11-14 06:35:09,796] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:44:01.1392793Z [1699943709.810672] [99f90b6e6ff3:10591:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1393498Z [1699943709.810672] [99f90b6e6ff3:10591:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1394257Z [1699943709.815899] [99f90b6e6ff3:10592:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1394927Z [1699943709.815899] [99f90b6e6ff3:10592:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1395101Z PASSED [6.2993s] [100%] 2023-11-14T06:44:01.1395122Z 2023-11-14T06:44:01.1396373Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15ac746e6903e41c.xml - 2023-11-14T06:44:01.1396657Z ============================== 1 passed in 6.34s =============================== 2023-11-14T06:44:01.1397677Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70cabe0e1683c323.xml 2023-11-14T06:44:01.1397946Z ============================= test session starts ============================== 2023-11-14T06:44:01.1398615Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1398771Z cachedir: .pytest_cache 2023-11-14T06:44:01.1399636Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1399816Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1399971Z configfile: pytest.ini 2023-11-14T06:44:01.1400784Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1400954Z collecting ... collected 1 item 2023-11-14T06:44:01.1402071Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2023-11-14T06:44:01.1402095Z 2023-11-14T06:44:01.1404158Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1405559Z [rank1]:[2023-11-14 06:35:21,467] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:44:01.1406965Z [rank0]:[2023-11-14 06:35:21,537] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:44:01.1407629Z [1699943721.548649] [99f90b6e6ff3:10618:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1408333Z [1699943721.548649] [99f90b6e6ff3:10618:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1408991Z [1699943721.548902] [99f90b6e6ff3:10619:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1409668Z [1699943721.548902] [99f90b6e6ff3:10619:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1409842Z PASSED [6.3118s] [100%] 2023-11-14T06:44:01.1409862Z 2023-11-14T06:44:01.1411289Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70cabe0e1683c323.xml - 2023-11-14T06:44:01.1411582Z ============================== 1 passed in 6.35s =============================== 2023-11-14T06:44:01.1412605Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2a80c8207521c2d.xml 2023-11-14T06:44:01.1412876Z ============================= test session starts ============================== 2023-11-14T06:44:01.1413688Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1413849Z cachedir: .pytest_cache 2023-11-14T06:44:01.1414694Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1414888Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1415043Z configfile: pytest.ini 2023-11-14T06:44:01.1415859Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1416031Z collecting ... collected 1 item 2023-11-14T06:44:01.1417076Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2023-11-14T06:44:01.1417099Z 2023-11-14T06:44:01.1419140Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1420548Z [rank0]:[2023-11-14 06:35:33,156] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:44:01.1421951Z [rank1]:[2023-11-14 06:35:33,209] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:44:01.1422608Z [1699943733.229107] [99f90b6e6ff3:10646:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1423402Z [1699943733.229107] [99f90b6e6ff3:10646:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1424091Z [1699943733.227217] [99f90b6e6ff3:10645:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1424822Z [1699943733.227217] [99f90b6e6ff3:10645:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1425002Z PASSED [6.4009s] [100%] 2023-11-14T06:44:01.1425022Z 2023-11-14T06:44:01.1426275Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2a80c8207521c2d.xml - 2023-11-14T06:44:01.1426539Z ============================== 1 passed in 6.44s =============================== 2023-11-14T06:44:01.1427576Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecc7c45f61279a66.xml 2023-11-14T06:44:01.1427847Z ============================= test session starts ============================== 2023-11-14T06:44:01.1428511Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1428677Z cachedir: .pytest_cache 2023-11-14T06:44:01.1429519Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1429719Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1429876Z configfile: pytest.ini 2023-11-14T06:44:01.1430671Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1430859Z collecting ... collected 1 item 2023-11-14T06:44:01.1431626Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2023-11-14T06:44:01.1431647Z 2023-11-14T06:44:01.1433420Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1434208Z [1699943744.923809] [99f90b6e6ff3:10672:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1434894Z [1699943744.923809] [99f90b6e6ff3:10672:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1438267Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:44:01.1438923Z [1699943744.927506] [99f90b6e6ff3:10673:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1439630Z [1699943744.927506] [99f90b6e6ff3:10673:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1442963Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:44:01.1443219Z PASSED [6.1997s] [100%] 2023-11-14T06:44:01.1443243Z 2023-11-14T06:44:01.1444513Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecc7c45f61279a66.xml - 2023-11-14T06:44:01.1444802Z ============================== 1 passed in 6.24s =============================== 2023-11-14T06:44:01.1445808Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26677226395f4519.xml 2023-11-14T06:44:01.1446076Z ============================= test session starts ============================== 2023-11-14T06:44:01.1446740Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1446898Z cachedir: .pytest_cache 2023-11-14T06:44:01.1447737Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1447940Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1448097Z configfile: pytest.ini 2023-11-14T06:44:01.1448914Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1449089Z collecting ... collected 1 item 2023-11-14T06:44:01.1449899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2023-11-14T06:44:01.1449919Z 2023-11-14T06:44:01.1451929Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1452601Z [1699943756.432818] [99f90b6e6ff3:10699:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1453311Z [1699943756.432818] [99f90b6e6ff3:10699:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1453980Z STAGE:2023-11-14 06:35:56 10699:10699 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1454784Z [1699943756.435737] [99f90b6e6ff3:10700:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1455480Z [1699943756.435737] [99f90b6e6ff3:10700:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1456141Z STAGE:2023-11-14 06:35:56 10700:10700 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1456809Z STAGE:2023-11-14 06:35:56 10699:10699 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1457489Z STAGE:2023-11-14 06:35:56 10700:10700 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1458959Z STAGE:2023-11-14 06:35:57 10699:10699 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:35:57 10700:10700 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1458981Z 2023-11-14T06:44:01.1462349Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:44:01.1465790Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:44:01.1466501Z STAGE:2023-11-14 06:35:57 10699:10699 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1467150Z STAGE:2023-11-14 06:35:57 10700:10700 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1467835Z STAGE:2023-11-14 06:35:57 10699:10699 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1469269Z STAGE:2023-11-14 06:35:57 10699:10699 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:35:57 10700:10700 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1469290Z 2023-11-14T06:44:01.1470022Z STAGE:2023-11-14 06:35:57 10700:10700 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1470174Z PASSED [6.5006s] [100%] 2023-11-14T06:44:01.1470194Z 2023-11-14T06:44:01.1471428Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26677226395f4519.xml - 2023-11-14T06:44:01.1471720Z ============================== 1 passed in 6.54s =============================== 2023-11-14T06:44:01.1472731Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd5aa89806a6e919.xml 2023-11-14T06:44:01.1472999Z ============================= test session starts ============================== 2023-11-14T06:44:01.1473669Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1473826Z cachedir: .pytest_cache 2023-11-14T06:44:01.1474694Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1474878Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1475032Z configfile: pytest.ini 2023-11-14T06:44:01.1475852Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1476139Z collecting ... collected 1 item 2023-11-14T06:44:01.1476984Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2023-11-14T06:44:01.1477026Z 2023-11-14T06:44:01.1478866Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1481734Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:2217: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-11-14T06:44:01.1481912Z warnings.warn( 2023-11-14T06:44:01.1484752Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:2217: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-11-14T06:44:01.1484919Z warnings.warn( 2023-11-14T06:44:01.1485579Z [1699943768.302302] [99f90b6e6ff3:10731:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1486386Z [1699943768.302302] [99f90b6e6ff3:10731:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1487071Z [1699943768.295259] [99f90b6e6ff3:10730:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1487747Z [1699943768.295259] [99f90b6e6ff3:10730:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1487923Z PASSED [6.3002s] [100%] 2023-11-14T06:44:01.1487944Z 2023-11-14T06:44:01.1489201Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd5aa89806a6e919.xml - 2023-11-14T06:44:01.1489468Z ============================== 1 passed in 6.34s =============================== 2023-11-14T06:44:01.1490503Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-357e4ebe59b1b54f.xml 2023-11-14T06:44:01.1490943Z ============================= test session starts ============================== 2023-11-14T06:44:01.1491628Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1491787Z cachedir: .pytest_cache 2023-11-14T06:44:01.1492638Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1492839Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1492993Z configfile: pytest.ini 2023-11-14T06:44:01.1493794Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1493986Z collecting ... collected 1 item 2023-11-14T06:44:01.1494792Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1494815Z 2023-11-14T06:44:01.1496630Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1497290Z [1699943779.976037] [99f90b6e6ff3:10758:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1498119Z [1699943779.976037] [99f90b6e6ff3:10758:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1498808Z STAGE:2023-11-14 06:36:20 10758:10758 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1499465Z [1699943779.970792] [99f90b6e6ff3:10757:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1500160Z [1699943779.970792] [99f90b6e6ff3:10757:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1500820Z STAGE:2023-11-14 06:36:20 10757:10757 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1501556Z STAGE:2023-11-14 06:36:20 10758:10758 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1502239Z STAGE:2023-11-14 06:36:20 10757:10757 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1502952Z STAGE:2023-11-14 06:36:20 10757:10757 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1503678Z STAGE:2023-11-14 06:36:20 10758:10758 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1504350Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1505119Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1506729Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1507492Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1508992Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1509578Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1510994Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1511725Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1513193Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1513962Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1515697Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1516529Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1518019Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1518910Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1520177Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1521062Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1522238Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1523099Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1523923Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1524470Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1525481Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1526632Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1527180Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1528223Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1529029Z [rank0]:[2023-11-14 06:36:20,668] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.1529786Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1530576Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1532383Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1533127Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1534608Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1535194Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1536817Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1537575Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1539036Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1539800Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1541531Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1542388Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1544015Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1544907Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1546061Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1546944Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1548142Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1548991Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1549806Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1550350Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1551304Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1552462Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1553095Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1554166Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1554976Z [rank1]:[2023-11-14 06:36:20,890] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.1555235Z ('RERUN', {'yellow': True}) [6.6107s] [100%] 2023-11-14T06:44:01.1557619Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [1699943786.362246] [99f90b6e6ff3:10784:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1558320Z [1699943786.362246] [99f90b6e6ff3:10784:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1558977Z STAGE:2023-11-14 06:36:26 10784:10784 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1559652Z [1699943786.364988] [99f90b6e6ff3:10785:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1560337Z [1699943786.364988] [99f90b6e6ff3:10785:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1561007Z STAGE:2023-11-14 06:36:26 10785:10785 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1561680Z STAGE:2023-11-14 06:36:26 10784:10784 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1562341Z STAGE:2023-11-14 06:36:26 10785:10785 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1563832Z STAGE:2023-11-14 06:36:26 10784:10784 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:36:26 10785:10785 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1563856Z 2023-11-14T06:44:01.1564525Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1565416Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1566887Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1567618Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1569094Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1569682Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1571326Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1572065Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1573549Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1574287Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1576131Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1577018Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1578497Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1579395Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1580527Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1581417Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1582611Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1583471Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1584306Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1584835Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1585788Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1586947Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1587476Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1588693Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1589482Z [rank1]:[2023-11-14 06:36:27,044] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.1590147Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1590933Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1592408Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1593163Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1594630Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1595215Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1596649Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1597363Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1598925Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1599697Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1601429Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1602283Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1603760Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1604666Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1605804Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1606716Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1607891Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1608740Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1609580Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1610108Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1612788Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1614254Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1614784Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1615850Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1616637Z [rank0]:[2023-11-14 06:36:27,242] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.1616927Z ('RERUN', {'yellow': True}) [6.4142s] [100%] 2023-11-14T06:44:01.1619298Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [1699943792.686359] [99f90b6e6ff3:10811:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1619998Z [1699943792.686359] [99f90b6e6ff3:10811:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1620673Z STAGE:2023-11-14 06:36:33 10811:10811 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1621330Z [1699943792.690017] [99f90b6e6ff3:10812:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1622136Z [1699943792.690017] [99f90b6e6ff3:10812:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1622814Z STAGE:2023-11-14 06:36:33 10812:10812 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1623486Z STAGE:2023-11-14 06:36:33 10811:10811 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1624176Z STAGE:2023-11-14 06:36:33 10812:10812 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1625744Z STAGE:2023-11-14 06:36:33 10811:10811 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:36:33 10812:10812 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1625770Z 2023-11-14T06:44:01.1626463Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1627232Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1628708Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1629471Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1630931Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1631535Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1632951Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1633669Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1635149Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1636003Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1637749Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1638584Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1640090Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1640976Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1642118Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1643021Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1644184Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1645045Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1645956Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1646518Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1647495Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1648632Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1649176Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1650226Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1651278Z [rank1]:[2023-11-14 06:36:33,365] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.1651985Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1652765Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1654251Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1654986Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1656443Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1657057Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1658477Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1659374Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1660828Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1661572Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1663322Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1664163Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1665673Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1666553Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1667704Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1668740Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1669939Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1670814Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1671632Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1672178Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1673138Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1674283Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1674833Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1675883Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1676693Z [rank0]:[2023-11-14 06:36:33,428] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.1676844Z FAILED [6.1138s] [100%] 2023-11-14T06:44:01.1676866Z 2023-11-14T06:44:01.1677108Z ==================================== RERUNS ==================================== 2023-11-14T06:44:01.1677544Z __________ TestDistBackendWithSpawn.test_ddp_sync_bn_training_vs_eval __________ 2023-11-14T06:44:01.1677721Z Traceback (most recent call last): 2023-11-14T06:44:01.1678642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.1678825Z self._join_processes(fn) 2023-11-14T06:44:01.1679784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.1680079Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.1681099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.1681265Z raise RuntimeError(error) 2023-11-14T06:44:01.1681626Z RuntimeError: Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.1681802Z Traceback (most recent call last): 2023-11-14T06:44:01.1682712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1682893Z getattr(self, test_name)() 2023-11-14T06:44:01.1683797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1683932Z fn() 2023-11-14T06:44:01.1684812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1684976Z method(*args, **kwargs) 2023-11-14T06:44:01.1685886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1686053Z return func(*args, **kwargs) 2023-11-14T06:44:01.1687222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1687461Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1688381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1688747Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1689260Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1689527Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1690059Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1690286Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1690492Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.1690513Z 2023-11-14T06:44:01.1691094Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1691692Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1691713Z 2023-11-14T06:44:01.1692172Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1692193Z 2023-11-14T06:44:01.1692232Z 2023-11-14T06:44:01.1692630Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1693027Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1693422Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.1693701Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.1694117Z __________ TestDistBackendWithSpawn.test_ddp_sync_bn_training_vs_eval __________ 2023-11-14T06:44:01.1694314Z Traceback (most recent call last): 2023-11-14T06:44:01.1695229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.1695391Z self._join_processes(fn) 2023-11-14T06:44:01.1696376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.1696571Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.1697590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.1697756Z raise RuntimeError(error) 2023-11-14T06:44:01.1698242Z RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.1698443Z Traceback (most recent call last): 2023-11-14T06:44:01.1699371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1699532Z getattr(self, test_name)() 2023-11-14T06:44:01.1700457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1700590Z fn() 2023-11-14T06:44:01.1701453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1701628Z method(*args, **kwargs) 2023-11-14T06:44:01.1702529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1702714Z return func(*args, **kwargs) 2023-11-14T06:44:01.1703890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1704114Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1705050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1705310Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1705787Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1706066Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1706580Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1706938Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1707166Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.1707187Z 2023-11-14T06:44:01.1707518Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1708127Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1708148Z 2023-11-14T06:44:01.1708558Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1708578Z 2023-11-14T06:44:01.1708844Z Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.1709020Z Traceback (most recent call last): 2023-11-14T06:44:01.1709941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1710124Z getattr(self, test_name)() 2023-11-14T06:44:01.1711034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1711168Z fn() 2023-11-14T06:44:01.1712048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1712212Z method(*args, **kwargs) 2023-11-14T06:44:01.1713111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1713295Z return func(*args, **kwargs) 2023-11-14T06:44:01.1714465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1714701Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1715612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1715871Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1716376Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1716635Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1717261Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1717505Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1717710Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.1717730Z 2023-11-14T06:44:01.1718081Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1718662Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1718682Z 2023-11-14T06:44:01.1719090Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1719112Z 2023-11-14T06:44:01.1719131Z 2023-11-14T06:44:01.1719535Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1719935Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1720338Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.1720617Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.1721004Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1721412Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1721659Z =================================== FAILURES =================================== 2023-11-14T06:44:01.1722073Z __________ TestDistBackendWithSpawn.test_ddp_sync_bn_training_vs_eval __________ 2023-11-14T06:44:01.1722271Z Traceback (most recent call last): 2023-11-14T06:44:01.1723183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.1723347Z self._join_processes(fn) 2023-11-14T06:44:01.1724416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.1724623Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.1725710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.1725876Z raise RuntimeError(error) 2023-11-14T06:44:01.1726218Z RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.1726416Z Traceback (most recent call last): 2023-11-14T06:44:01.1727327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1727491Z getattr(self, test_name)() 2023-11-14T06:44:01.1728405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1728540Z fn() 2023-11-14T06:44:01.1729437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1729594Z method(*args, **kwargs) 2023-11-14T06:44:01.1730499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1730906Z return func(*args, **kwargs) 2023-11-14T06:44:01.1732107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1732324Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1733262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1733522Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1734024Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1734288Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1734801Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1735200Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1735408Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.1735429Z 2023-11-14T06:44:01.1735759Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1736362Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1736383Z 2023-11-14T06:44:01.1736794Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1736814Z 2023-11-14T06:44:01.1736833Z 2023-11-14T06:44:01.1737237Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1737632Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1738021Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.1738313Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.1738703Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1739100Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1739497Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1739885Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1741152Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-357e4ebe59b1b54f.xml - 2023-11-14T06:44:01.1741427Z =========================== short test summary info ============================ 2023-11-14T06:44:01.1742781Z FAILED [6.1138s] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval - RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.1742997Z Traceback (most recent call last): 2023-11-14T06:44:01.1743933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1744099Z getattr(self, test_name)() 2023-11-14T06:44:01.1745020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1745152Z fn() 2023-11-14T06:44:01.1746032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1746186Z method(*args, **kwargs) 2023-11-14T06:44:01.1747078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1747264Z return func(*args, **kwargs) 2023-11-14T06:44:01.1748447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1748673Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1749607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1749868Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1750361Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1750618Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1751129Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1751373Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1751605Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.1751627Z 2023-11-14T06:44:01.1751961Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1752560Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1752671Z 2023-11-14T06:44:01.1753091Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1753397Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2023-11-14T06:44:01.1753675Z ========================= 1 failed, 2 rerun in 19.18s ========================== 2023-11-14T06:44:01.1753878Z Got exit code 1, retrying (retries left=1) 2023-11-14T06:44:01.1754917Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c3e54f9d9264c85.xml 2023-11-14T06:44:01.1755190Z ============================= test session starts ============================== 2023-11-14T06:44:01.1755837Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.1756013Z cachedir: .pytest_cache 2023-11-14T06:44:01.1756858Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.1757059Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.1757213Z configfile: pytest.ini 2023-11-14T06:44:01.1758016Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.1758206Z collecting ... collected 1 item 2023-11-14T06:44:01.1759012Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1759032Z 2023-11-14T06:44:01.1760906Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.1761606Z [1699943804.544200] [99f90b6e6ff3:10843:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1762295Z [1699943804.544200] [99f90b6e6ff3:10843:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1762983Z STAGE:2023-11-14 06:36:45 10843:10843 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1763637Z [1699943804.538884] [99f90b6e6ff3:10842:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1764318Z [1699943804.538884] [99f90b6e6ff3:10842:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1764994Z STAGE:2023-11-14 06:36:45 10842:10842 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1765672Z STAGE:2023-11-14 06:36:45 10843:10843 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1766356Z STAGE:2023-11-14 06:36:45 10842:10842 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1767061Z STAGE:2023-11-14 06:36:45 10842:10842 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1767771Z STAGE:2023-11-14 06:36:45 10843:10843 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1768460Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1769232Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1770950Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1771722Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1773184Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1773942Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1775361Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1776115Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1777590Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1778337Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1780092Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1780926Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1782423Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1783410Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1784578Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1785488Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1786655Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1787513Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1788330Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1788863Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1789841Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1790983Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1791530Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1792575Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1793363Z [rank0]:[2023-11-14 06:36:45,241] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.1794042Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1794818Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1796304Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1797143Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1798602Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1799205Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1800627Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1801362Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1802828Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1803589Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1805317Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1806258Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1807778Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1808667Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1809825Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1810876Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1812077Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1812941Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1813765Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1814309Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1815269Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1816408Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1816955Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1818007Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1818815Z [rank1]:[2023-11-14 06:36:45,625] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.1819268Z ('RERUN', {'yellow': True}) [6.8056s] [100%] 2023-11-14T06:44:01.1821634Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [1699943811.139340] [99f90b6e6ff3:10869:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1822343Z [1699943811.139340] [99f90b6e6ff3:10869:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1823009Z STAGE:2023-11-14 06:36:51 10869:10869 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1823682Z [1699943811.141530] [99f90b6e6ff3:10870:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1824368Z [1699943811.141530] [99f90b6e6ff3:10870:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1825071Z STAGE:2023-11-14 06:36:51 10870:10870 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1825768Z STAGE:2023-11-14 06:36:51 10870:10870 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1826431Z STAGE:2023-11-14 06:36:51 10869:10869 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1827153Z STAGE:2023-11-14 06:36:51 10870:10870 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1827860Z STAGE:2023-11-14 06:36:51 10869:10869 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1828642Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1829457Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1830939Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1831689Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1833145Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1833734Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1835177Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1835903Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1837377Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1838121Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1839854Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1840707Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1842289Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1843186Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1844320Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1845225Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1846403Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1847261Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1848098Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1848625Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1849581Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1850964Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1851627Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1852715Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1853513Z [rank0]:[2023-11-14 06:36:51,830] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.1854176Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1854965Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1856435Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1857190Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1858647Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1859261Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1860676Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1861388Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1862872Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1863620Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1865499Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1866332Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1867806Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1868711Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1869849Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1870759Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1871928Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1872775Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1873609Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1874224Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1875225Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1876373Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1876901Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1877963Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1878748Z [rank1]:[2023-11-14 06:36:51,934] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.1879026Z ('RERUN', {'yellow': True}) [6.3139s] [100%] 2023-11-14T06:44:01.1881395Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [1699943817.410119] [99f90b6e6ff3:10896:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1882090Z [1699943817.410119] [99f90b6e6ff3:10896:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1882765Z STAGE:2023-11-14 06:36:57 10896:10896 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1883419Z [1699943817.417862] [99f90b6e6ff3:10897:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.1884117Z [1699943817.417862] [99f90b6e6ff3:10897:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.1884779Z STAGE:2023-11-14 06:36:57 10897:10897 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.1885451Z STAGE:2023-11-14 06:36:58 10896:10896 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1886130Z STAGE:2023-11-14 06:36:58 10897:10897 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.1887725Z STAGE:2023-11-14 06:36:58 10897:10897 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:36:58 10896:10896 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.1887749Z 2023-11-14T06:44:01.1888435Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1889203Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1890906Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1891690Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1893157Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1893760Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1895176Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1895907Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1897503Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1898287Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1900035Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1900866Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1902353Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1903245Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1904378Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1905288Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1906459Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1907322Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1908144Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1908671Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1909643Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1910916Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1911461Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1912514Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1913305Z [rank1]:[2023-11-14 06:36:58,092] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:44:01.1913992Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:44:01.1914761Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:44:01.1916258Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1916988Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:44:01.1918441Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1919047Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:44:01.1920546Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1921303Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:44:01.1922763Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1923521Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:44:01.1925308Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1926149Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1927647Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1928532Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1929683Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1930569Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1931976Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1932992Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1933812Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] AssertionError: AssertionError not raised 2023-11-14T06:44:01.1934354Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1935313Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1936456Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1937007Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:44:01.1938064Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1938874Z [rank0]:[2023-11-14 06:36:58,238] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10 2023-11-14T06:44:01.1939027Z FAILED [6.2138s] [100%] 2023-11-14T06:44:01.1939048Z 2023-11-14T06:44:01.1939290Z ==================================== RERUNS ==================================== 2023-11-14T06:44:01.1939722Z __________ TestDistBackendWithSpawn.test_ddp_sync_bn_training_vs_eval __________ 2023-11-14T06:44:01.1939899Z Traceback (most recent call last): 2023-11-14T06:44:01.1940812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.1940992Z self._join_processes(fn) 2023-11-14T06:44:01.1942063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.1942291Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.1943304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.1943470Z raise RuntimeError(error) 2023-11-14T06:44:01.1943828Z RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.1944006Z Traceback (most recent call last): 2023-11-14T06:44:01.1944921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1945100Z getattr(self, test_name)() 2023-11-14T06:44:01.1945995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1946146Z fn() 2023-11-14T06:44:01.1947010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1947169Z method(*args, **kwargs) 2023-11-14T06:44:01.1948085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1948249Z return func(*args, **kwargs) 2023-11-14T06:44:01.1949425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1949661Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1950575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1950852Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1951336Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1951594Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1952124Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1952450Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1952655Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.1952678Z 2023-11-14T06:44:01.1953024Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1953606Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1953626Z 2023-11-14T06:44:01.1954051Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1954072Z 2023-11-14T06:44:01.1954319Z Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.1954494Z Traceback (most recent call last): 2023-11-14T06:44:01.1955434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1955595Z getattr(self, test_name)() 2023-11-14T06:44:01.1956496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1956652Z fn() 2023-11-14T06:44:01.1957516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1957670Z method(*args, **kwargs) 2023-11-14T06:44:01.1958581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1958746Z return func(*args, **kwargs) 2023-11-14T06:44:01.1959942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1960256Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1961192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1961476Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1961955Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1962212Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1962739Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1962968Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1963191Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.1963211Z 2023-11-14T06:44:01.1963539Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1964118Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1964139Z 2023-11-14T06:44:01.1964575Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1964595Z 2023-11-14T06:44:01.1964614Z 2023-11-14T06:44:01.1965000Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1965420Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1965798Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.1966067Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.1966502Z __________ TestDistBackendWithSpawn.test_ddp_sync_bn_training_vs_eval __________ 2023-11-14T06:44:01.1966678Z Traceback (most recent call last): 2023-11-14T06:44:01.1967584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.1967762Z self._join_processes(fn) 2023-11-14T06:44:01.1968728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.1968944Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.1969938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.1970203Z raise RuntimeError(error) 2023-11-14T06:44:01.1970562Z RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.1970960Z Traceback (most recent call last): 2023-11-14T06:44:01.1971896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1972074Z getattr(self, test_name)() 2023-11-14T06:44:01.1972971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1973103Z fn() 2023-11-14T06:44:01.1973986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1974142Z method(*args, **kwargs) 2023-11-14T06:44:01.1975059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1975232Z return func(*args, **kwargs) 2023-11-14T06:44:01.1976409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1976644Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1977560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1977818Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1978311Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1978694Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1979244Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1979476Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1979680Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.1979701Z 2023-11-14T06:44:01.1980048Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1980631Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1980652Z 2023-11-14T06:44:01.1981062Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1981082Z 2023-11-14T06:44:01.1981346Z Process 1 exited with error code 10 and exception: 2023-11-14T06:44:01.1981522Z Traceback (most recent call last): 2023-11-14T06:44:01.1982465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.1982626Z getattr(self, test_name)() 2023-11-14T06:44:01.1983525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.1983683Z fn() 2023-11-14T06:44:01.1984543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.1984698Z method(*args, **kwargs) 2023-11-14T06:44:01.1985615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.1985780Z return func(*args, **kwargs) 2023-11-14T06:44:01.1986970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1987186Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.1988105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.1988386Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.1989033Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.1989290Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.1989824Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.1990049Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.1990253Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.1990292Z 2023-11-14T06:44:01.1990620Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.1991205Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.1991225Z 2023-11-14T06:44:01.1991657Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.1991678Z 2023-11-14T06:44:01.1991696Z 2023-11-14T06:44:01.1992147Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1992553Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1992955Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.1993223Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.1993623Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.1994017Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.1994260Z =================================== FAILURES =================================== 2023-11-14T06:44:01.1994688Z __________ TestDistBackendWithSpawn.test_ddp_sync_bn_training_vs_eval __________ 2023-11-14T06:44:01.1994865Z Traceback (most recent call last): 2023-11-14T06:44:01.1995864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:44:01.1996060Z self._join_processes(fn) 2023-11-14T06:44:01.1997037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes 2023-11-14T06:44:01.1997250Z self._check_return_codes(elapsed_time) 2023-11-14T06:44:01.1998241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 811, in _check_return_codes 2023-11-14T06:44:01.1998405Z raise RuntimeError(error) 2023-11-14T06:44:01.1998765Z RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.1998941Z Traceback (most recent call last): 2023-11-14T06:44:01.1999853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.2000037Z getattr(self, test_name)() 2023-11-14T06:44:01.2000945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.2001078Z fn() 2023-11-14T06:44:01.2001964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.2002120Z method(*args, **kwargs) 2023-11-14T06:44:01.2003041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.2003208Z return func(*args, **kwargs) 2023-11-14T06:44:01.2004379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.2004613Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.2005531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.2005793Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.2006286Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.2006643Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.2007175Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.2007403Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.2007608Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.2007629Z 2023-11-14T06:44:01.2007973Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.2008553Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.2008574Z 2023-11-14T06:44:01.2008979Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.2009018Z 2023-11-14T06:44:01.2009037Z 2023-11-14T06:44:01.2009428Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.2009822Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.2010228Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:44:01.2010495Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:44:01.2011108Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.2011527Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.2011910Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:44:01.2012295Z Process 0 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:44:01.2013558Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c3e54f9d9264c85.xml - 2023-11-14T06:44:01.2013951Z =========================== short test summary info ============================ 2023-11-14T06:44:01.2015235Z FAILED [6.2138s] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval - RuntimeError: Process 0 exited with error code 10 and exception: 2023-11-14T06:44:01.2015420Z Traceback (most recent call last): 2023-11-14T06:44:01.2016336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:44:01.2016514Z getattr(self, test_name)() 2023-11-14T06:44:01.2017416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:44:01.2017565Z fn() 2023-11-14T06:44:01.2018426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:44:01.2018581Z method(*args, **kwargs) 2023-11-14T06:44:01.2019497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:44:01.2019662Z return func(*args, **kwargs) 2023-11-14T06:44:01.2020846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9230, in test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.2021077Z self.assertNotEqual([], all_gather_calls) 2023-11-14T06:44:01.2021992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in assertNotEqual 2023-11-14T06:44:01.2022267Z with self.assertRaises(AssertionError, msg=msg): 2023-11-14T06:44:01.2022744Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in __exit__ 2023-11-14T06:44:01.2023000Z self._raiseFailure("{} not raised".format(exc_name)) 2023-11-14T06:44:01.2023538Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure 2023-11-14T06:44:01.2023763Z raise self.test_case.failureException(msg) 2023-11-14T06:44:01.2023968Z AssertionError: AssertionError not raised 2023-11-14T06:44:01.2024113Z 2023-11-14T06:44:01.2024474Z To execute this test, run the following from the base repo dir: 2023-11-14T06:44:01.2025102Z python test/distributed/test_distributed_spawn.py -k test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.2025123Z 2023-11-14T06:44:01.2025548Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:44:01.2025834Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2023-11-14T06:44:01.2026114Z ========================= 1 failed, 2 rerun in 19.38s ========================== 2023-11-14T06:44:01.2029140Z Test exited with non-zero exitcode 1. Command to reproduce: BACKEND=ucc WORLD_SIZE=2 /opt/conda/envs/py_3.10/bin/python distributed/test_distributed_spawn.py --import-disabled-tests --import-slow-tests --use-pytest --shard-id=0 --num-shards=2 -v -vv -rfEX -p no:xdist -x --reruns=2 --pytest-single-test distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.2030164Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e38ececbc8461d34.xml 2023-11-14T06:44:01.2030450Z ============================= test session starts ============================== 2023-11-14T06:44:01.2031091Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2031247Z cachedir: .pytest_cache 2023-11-14T06:44:01.2032111Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2032287Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2032439Z configfile: pytest.ini 2023-11-14T06:44:01.2033340Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2033521Z collecting ... collected 1 item 2023-11-14T06:44:01.2034317Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2023-11-14T06:44:01.2034362Z 2023-11-14T06:44:01.2036158Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2036818Z [1699943829.198298] [99f90b6e6ff3:10927:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2037519Z [1699943829.198298] [99f90b6e6ff3:10927:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2038275Z [1699943829.199857] [99f90b6e6ff3:10928:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2038965Z [1699943829.199857] [99f90b6e6ff3:10928:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2039123Z PASSED [6.0996s] [100%] 2023-11-14T06:44:01.2039143Z 2023-11-14T06:44:01.2040400Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e38ececbc8461d34.xml - 2023-11-14T06:44:01.2040680Z ============================== 1 passed in 6.14s =============================== 2023-11-14T06:44:01.2041691Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e51b7760cd8b58e.xml 2023-11-14T06:44:01.2041979Z ============================= test session starts ============================== 2023-11-14T06:44:01.2042623Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2042784Z cachedir: .pytest_cache 2023-11-14T06:44:01.2043648Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2043928Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2044081Z configfile: pytest.ini 2023-11-14T06:44:01.2044905Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2045073Z collecting ... collected 1 item 2023-11-14T06:44:01.2045814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2023-11-14T06:44:01.2045853Z 2023-11-14T06:44:01.2047582Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2048053Z SKIPPED [0.2932s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [100%] 2023-11-14T06:44:01.2048074Z 2023-11-14T06:44:01.2049334Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e51b7760cd8b58e.xml - 2023-11-14T06:44:01.2049609Z ============================== 1 skipped in 0.33s ============================== 2023-11-14T06:44:01.2050635Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07d5c30aa0d94c3e.xml 2023-11-14T06:44:01.2051140Z ============================= test session starts ============================== 2023-11-14T06:44:01.2051796Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2051968Z cachedir: .pytest_cache 2023-11-14T06:44:01.2052809Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2053107Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2053300Z configfile: pytest.ini 2023-11-14T06:44:01.2054106Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2054280Z collecting ... collected 1 item 2023-11-14T06:44:01.2055202Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2023-11-14T06:44:01.2055223Z 2023-11-14T06:44:01.2057118Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2057795Z [1699943846.354206] [99f90b6e6ff3:10953:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2058489Z [1699943846.354206] [99f90b6e6ff3:10953:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2059157Z [1699943846.363903] [99f90b6e6ff3:10954:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2059832Z [1699943846.363903] [99f90b6e6ff3:10954:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2059981Z PASSED [6.3015s] [100%] 2023-11-14T06:44:01.2060001Z 2023-11-14T06:44:01.2061266Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07d5c30aa0d94c3e.xml - 2023-11-14T06:44:01.2061529Z ============================== 1 passed in 6.34s =============================== 2023-11-14T06:44:01.2062566Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e7b1edb63b95024.xml 2023-11-14T06:44:01.2062841Z ============================= test session starts ============================== 2023-11-14T06:44:01.2063485Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2063783Z cachedir: .pytest_cache 2023-11-14T06:44:01.2064635Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2064810Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2064980Z configfile: pytest.ini 2023-11-14T06:44:01.2065779Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2065948Z collecting ... collected 1 item 2023-11-14T06:44:01.2075098Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2023-11-14T06:44:01.2075130Z 2023-11-14T06:44:01.2077021Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2078021Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/init.py:421: UserWarning: Initializing zero-element tensors is a no-op 2023-11-14T06:44:01.2078421Z warnings.warn("Initializing zero-element tensors is a no-op") 2023-11-14T06:44:01.2079390Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/init.py:421: UserWarning: Initializing zero-element tensors is a no-op 2023-11-14T06:44:01.2079783Z warnings.warn("Initializing zero-element tensors is a no-op") 2023-11-14T06:44:01.2080442Z [1699943857.994861] [99f90b6e6ff3:10980:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2081285Z [1699943857.994861] [99f90b6e6ff3:10980:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2081959Z [1699943857.997104] [99f90b6e6ff3:10981:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2082665Z [1699943857.997104] [99f90b6e6ff3:10981:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2082819Z PASSED [6.1023s] [100%] 2023-11-14T06:44:01.2082841Z 2023-11-14T06:44:01.2084108Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e7b1edb63b95024.xml - 2023-11-14T06:44:01.2084395Z ============================== 1 passed in 6.14s =============================== 2023-11-14T06:44:01.2085408Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ff41484c374238d.xml 2023-11-14T06:44:01.2085680Z ============================= test session starts ============================== 2023-11-14T06:44:01.2086352Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2086510Z cachedir: .pytest_cache 2023-11-14T06:44:01.2087379Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2087557Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2087711Z configfile: pytest.ini 2023-11-14T06:44:01.2088534Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2088704Z collecting ... collected 1 item 2023-11-14T06:44:01.2089452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2023-11-14T06:44:01.2089474Z 2023-11-14T06:44:01.2091574Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2091740Z PASSED [5.6021s] [100%] 2023-11-14T06:44:01.2091929Z 2023-11-14T06:44:01.2093230Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ff41484c374238d.xml - 2023-11-14T06:44:01.2093494Z ============================== 1 passed in 5.64s =============================== 2023-11-14T06:44:01.2094499Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-928dcf467aaabf46.xml 2023-11-14T06:44:01.2094793Z ============================= test session starts ============================== 2023-11-14T06:44:01.2095439Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2095616Z cachedir: .pytest_cache 2023-11-14T06:44:01.2096469Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2096648Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2096829Z configfile: pytest.ini 2023-11-14T06:44:01.2097630Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2097800Z collecting ... collected 1 item 2023-11-14T06:44:01.2098632Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2023-11-14T06:44:01.2098654Z 2023-11-14T06:44:01.2100455Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2101259Z [1699943880.416301] [99f90b6e6ff3:11015:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2101967Z [1699943880.416301] [99f90b6e6ff3:11015:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2102620Z [1699943880.419745] [99f90b6e6ff3:11016:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2103315Z [1699943880.419745] [99f90b6e6ff3:11016:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2106676Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:44:01.2110032Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:44:01.2110194Z PASSED [6.2113s] [100%] 2023-11-14T06:44:01.2110214Z 2023-11-14T06:44:01.2111494Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-928dcf467aaabf46.xml - 2023-11-14T06:44:01.2111760Z ============================== 1 passed in 6.25s =============================== 2023-11-14T06:44:01.2112796Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3445391f109a2985.xml 2023-11-14T06:44:01.2113169Z ============================= test session starts ============================== 2023-11-14T06:44:01.2113825Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2114001Z cachedir: .pytest_cache 2023-11-14T06:44:01.2114845Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2115022Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2115194Z configfile: pytest.ini 2023-11-14T06:44:01.2115997Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2116183Z collecting ... collected 1 item 2023-11-14T06:44:01.2117000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2023-11-14T06:44:01.2117022Z 2023-11-14T06:44:01.2118818Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2119507Z [1699943891.881762] [99f90b6e6ff3:11042:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2120191Z [1699943891.881762] [99f90b6e6ff3:11042:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2123669Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:44:01.2124352Z [1699943891.881989] [99f90b6e6ff3:11043:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2125097Z [1699943891.881989] [99f90b6e6ff3:11043:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2125249Z PASSED [6.2007s] [100%] 2023-11-14T06:44:01.2125269Z 2023-11-14T06:44:01.2126516Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3445391f109a2985.xml - 2023-11-14T06:44:01.2126800Z ============================== 1 passed in 6.24s =============================== 2023-11-14T06:44:01.2127819Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd990106e5ea4f4c.xml 2023-11-14T06:44:01.2128109Z ============================= test session starts ============================== 2023-11-14T06:44:01.2128759Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2128916Z cachedir: .pytest_cache 2023-11-14T06:44:01.2129772Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2129949Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2130102Z configfile: pytest.ini 2023-11-14T06:44:01.2131180Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2131361Z collecting ... collected 1 item 2023-11-14T06:44:01.2132194Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2023-11-14T06:44:01.2132215Z 2023-11-14T06:44:01.2134004Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2134315Z PASSED [5.6105s] [100%] 2023-11-14T06:44:01.2134335Z 2023-11-14T06:44:01.2135601Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd990106e5ea4f4c.xml - 2023-11-14T06:44:01.2135866Z ============================== 1 passed in 5.65s =============================== 2023-11-14T06:44:01.2136907Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a6db6a55c1eb027.xml 2023-11-14T06:44:01.2137184Z ============================= test session starts ============================== 2023-11-14T06:44:01.2137828Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2138004Z cachedir: .pytest_cache 2023-11-14T06:44:01.2138847Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2139026Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2139200Z configfile: pytest.ini 2023-11-14T06:44:01.2140001Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2140188Z collecting ... collected 1 item 2023-11-14T06:44:01.2140900Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2023-11-14T06:44:01.2140920Z 2023-11-14T06:44:01.2142976Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (CPU tensor ops not supported by UCP TL) [100%] 2023-11-14T06:44:01.2143002Z 2023-11-14T06:44:01.2144292Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a6db6a55c1eb027.xml - 2023-11-14T06:44:01.2144561Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2145588Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20f9b8518054c9ae.xml 2023-11-14T06:44:01.2145857Z ============================= test session starts ============================== 2023-11-14T06:44:01.2146498Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2146674Z cachedir: .pytest_cache 2023-11-14T06:44:01.2147520Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2147699Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2147871Z configfile: pytest.ini 2023-11-14T06:44:01.2148678Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2148865Z collecting ... collected 1 item 2023-11-14T06:44:01.2149574Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2023-11-14T06:44:01.2149595Z 2023-11-14T06:44:01.2151487Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (CPU tensor ops not supported by UCP TL) [100%] 2023-11-14T06:44:01.2151526Z 2023-11-14T06:44:01.2152770Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20f9b8518054c9ae.xml - 2023-11-14T06:44:01.2153036Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2154173Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-68e37e40a3795350.xml 2023-11-14T06:44:01.2154440Z ============================= test session starts ============================== 2023-11-14T06:44:01.2155081Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2155257Z cachedir: .pytest_cache 2023-11-14T06:44:01.2156089Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2156286Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2156440Z configfile: pytest.ini 2023-11-14T06:44:01.2157237Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2157425Z collecting ... collected 1 item 2023-11-14T06:44:01.2158137Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2023-11-14T06:44:01.2158164Z 2023-11-14T06:44:01.2160067Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (CPU tensor ops not supported by UCP TL) [100%] 2023-11-14T06:44:01.2160109Z 2023-11-14T06:44:01.2161343Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-68e37e40a3795350.xml - 2023-11-14T06:44:01.2161605Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2162715Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6b62e238ead2e1c.xml 2023-11-14T06:44:01.2162996Z ============================= test session starts ============================== 2023-11-14T06:44:01.2163648Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2163824Z cachedir: .pytest_cache 2023-11-14T06:44:01.2164665Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2164858Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2165013Z configfile: pytest.ini 2023-11-14T06:44:01.2165812Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2166002Z collecting ... collected 1 item 2023-11-14T06:44:01.2166701Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2023-11-14T06:44:01.2166728Z 2023-11-14T06:44:01.2168416Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2168596Z PASSED [5.6098s] [100%] 2023-11-14T06:44:01.2168616Z 2023-11-14T06:44:01.2169854Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6b62e238ead2e1c.xml - 2023-11-14T06:44:01.2170137Z ============================== 1 passed in 5.65s =============================== 2023-11-14T06:44:01.2171403Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ece7483ad22102e.xml 2023-11-14T06:44:01.2171679Z ============================= test session starts ============================== 2023-11-14T06:44:01.2172356Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2172513Z cachedir: .pytest_cache 2023-11-14T06:44:01.2173374Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2173702Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2173857Z configfile: pytest.ini 2023-11-14T06:44:01.2174679Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2174848Z collecting ... collected 1 item 2023-11-14T06:44:01.2175611Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2023-11-14T06:44:01.2175631Z 2023-11-14T06:44:01.2177403Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2178063Z [1699943941.251947] [99f90b6e6ff3:11102:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2178770Z [1699943941.251947] [99f90b6e6ff3:11102:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2179419Z [1699943941.255393] [99f90b6e6ff3:11103:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2180087Z [1699943941.255393] [99f90b6e6ff3:11103:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2180257Z PASSED [6.2100s] [100%] 2023-11-14T06:44:01.2180277Z 2023-11-14T06:44:01.2181523Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ece7483ad22102e.xml - 2023-11-14T06:44:01.2181913Z ============================== 1 passed in 6.25s =============================== 2023-11-14T06:44:01.2182952Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c6fd4cfb1405d80.xml 2023-11-14T06:44:01.2183227Z ============================= test session starts ============================== 2023-11-14T06:44:01.2183891Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2184046Z cachedir: .pytest_cache 2023-11-14T06:44:01.2184909Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2185087Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2185241Z configfile: pytest.ini 2023-11-14T06:44:01.2186059Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2186228Z collecting ... collected 1 item 2023-11-14T06:44:01.2186995Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2023-11-14T06:44:01.2187016Z 2023-11-14T06:44:01.2188784Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2189442Z STAGE:2023-11-14 06:39:12 11130:11130 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2190101Z STAGE:2023-11-14 06:39:12 11129:11129 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2190688Z [1699943952.650310] [99f90b6e6ff3:11129:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2191264Z [1699943952.670578] [99f90b6e6ff3:11130:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2191947Z [1699943952.772938] [99f90b6e6ff3:11130:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2192630Z [1699943952.772938] [99f90b6e6ff3:11130:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2193446Z STAGE:2023-11-14 06:39:13 11130:11130 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2194156Z STAGE:2023-11-14 06:39:13 11130:11130 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2194809Z [1699943952.770429] [99f90b6e6ff3:11129:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2195508Z [1699943952.770429] [99f90b6e6ff3:11129:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2196184Z STAGE:2023-11-14 06:39:13 11129:11129 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2196900Z STAGE:2023-11-14 06:39:13 11129:11129 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2197071Z PASSED [6.4117s] [100%] 2023-11-14T06:44:01.2197097Z 2023-11-14T06:44:01.2198343Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c6fd4cfb1405d80.xml - 2023-11-14T06:44:01.2198626Z ============================== 1 passed in 6.45s =============================== 2023-11-14T06:44:01.2199634Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4979ea01a9c11f0.xml 2023-11-14T06:44:01.2199903Z ============================= test session starts ============================== 2023-11-14T06:44:01.2200560Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2200716Z cachedir: .pytest_cache 2023-11-14T06:44:01.2201658Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2201850Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2202011Z configfile: pytest.ini 2023-11-14T06:44:01.2202838Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2203007Z collecting ... collected 1 item 2023-11-14T06:44:01.2203840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2023-11-14T06:44:01.2203861Z 2023-11-14T06:44:01.2205956Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:01.2205978Z 2023-11-14T06:44:01.2207224Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4979ea01a9c11f0.xml - 2023-11-14T06:44:01.2207511Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2208535Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3cb4ddb8310cca42.xml 2023-11-14T06:44:01.2208822Z ============================= test session starts ============================== 2023-11-14T06:44:01.2209466Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2209623Z cachedir: .pytest_cache 2023-11-14T06:44:01.2210483Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2210661Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2211026Z configfile: pytest.ini 2023-11-14T06:44:01.2211856Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2212025Z collecting ... collected 1 item 2023-11-14T06:44:01.2213028Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2023-11-14T06:44:01.2213067Z 2023-11-14T06:44:01.2215182Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:01.2215204Z 2023-11-14T06:44:01.2216453Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3cb4ddb8310cca42.xml - 2023-11-14T06:44:01.2216742Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2217756Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-77919cd5b3a682ab.xml 2023-11-14T06:44:01.2218052Z ============================= test session starts ============================== 2023-11-14T06:44:01.2218697Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2218852Z cachedir: .pytest_cache 2023-11-14T06:44:01.2219714Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2219889Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2220043Z configfile: pytest.ini 2023-11-14T06:44:01.2220857Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2221026Z collecting ... collected 1 item 2023-11-14T06:44:01.2221980Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2023-11-14T06:44:01.2222005Z 2023-11-14T06:44:01.2224116Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:01.2224137Z 2023-11-14T06:44:01.2225426Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-77919cd5b3a682ab.xml - 2023-11-14T06:44:01.2225715Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2226723Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa61e37fde92b5ea.xml 2023-11-14T06:44:01.2227015Z ============================= test session starts ============================== 2023-11-14T06:44:01.2227664Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2227827Z cachedir: .pytest_cache 2023-11-14T06:44:01.2228686Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2228864Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2229018Z configfile: pytest.ini 2023-11-14T06:44:01.2229832Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2230001Z collecting ... collected 1 item 2023-11-14T06:44:01.2230818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2023-11-14T06:44:01.2230839Z 2023-11-14T06:44:01.2232898Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires backend ucc to be one of {'nccl'}) [100%] 2023-11-14T06:44:01.2233009Z 2023-11-14T06:44:01.2234298Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa61e37fde92b5ea.xml - 2023-11-14T06:44:01.2234565Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2235576Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f01c9b4c3e413a8.xml 2023-11-14T06:44:01.2235864Z ============================= test session starts ============================== 2023-11-14T06:44:01.2236501Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2236663Z cachedir: .pytest_cache 2023-11-14T06:44:01.2237525Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2237711Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2237886Z configfile: pytest.ini 2023-11-14T06:44:01.2238685Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2238856Z collecting ... collected 1 item 2023-11-14T06:44:01.2239675Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2023-11-14T06:44:01.2239696Z 2023-11-14T06:44:01.2241837Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires backend ucc to be one of {'nccl'}) [100%] 2023-11-14T06:44:01.2241861Z 2023-11-14T06:44:01.2243144Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f01c9b4c3e413a8.xml - 2023-11-14T06:44:01.2243418Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2244433Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f96bef79607a977e.xml 2023-11-14T06:44:01.2244721Z ============================= test session starts ============================== 2023-11-14T06:44:01.2245364Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2245519Z cachedir: .pytest_cache 2023-11-14T06:44:01.2246375Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2246560Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2246734Z configfile: pytest.ini 2023-11-14T06:44:01.2247533Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2247711Z collecting ... collected 1 item 2023-11-14T06:44:01.2248440Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2023-11-14T06:44:01.2248460Z 2023-11-14T06:44:01.2250296Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires world size of 4) [100%] 2023-11-14T06:44:01.2250317Z 2023-11-14T06:44:01.2251814Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f96bef79607a977e.xml - 2023-11-14T06:44:01.2252092Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2253105Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a546b61e64abc73.xml 2023-11-14T06:44:01.2253533Z ============================= test session starts ============================== 2023-11-14T06:44:01.2254187Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2254364Z cachedir: .pytest_cache 2023-11-14T06:44:01.2255206Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2255384Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2255555Z configfile: pytest.ini 2023-11-14T06:44:01.2256356Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2256524Z collecting ... collected 1 item 2023-11-14T06:44:01.2257358Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2023-11-14T06:44:01.2257379Z 2023-11-14T06:44:01.2259323Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires world size of 4) [100%] 2023-11-14T06:44:01.2259344Z 2023-11-14T06:44:01.2260604Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a546b61e64abc73.xml - 2023-11-14T06:44:01.2260870Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2261880Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9f20b49cd006029.xml 2023-11-14T06:44:01.2262278Z ============================= test session starts ============================== 2023-11-14T06:44:01.2262950Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2263132Z cachedir: .pytest_cache 2023-11-14T06:44:01.2263973Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2264149Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2264319Z configfile: pytest.ini 2023-11-14T06:44:01.2265117Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2265287Z collecting ... collected 1 item 2023-11-14T06:44:01.2266232Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2023-11-14T06:44:01.2266254Z 2023-11-14T06:44:01.2268175Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2268356Z PASSED [5.6113s] [100%] 2023-11-14T06:44:01.2268375Z 2023-11-14T06:44:01.2269614Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9f20b49cd006029.xml - 2023-11-14T06:44:01.2269880Z ============================== 1 passed in 5.65s =============================== 2023-11-14T06:44:01.2270903Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-846941ae5d294aea.xml 2023-11-14T06:44:01.2271172Z ============================= test session starts ============================== 2023-11-14T06:44:01.2271837Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2271994Z cachedir: .pytest_cache 2023-11-14T06:44:01.2272832Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2273129Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2273281Z configfile: pytest.ini 2023-11-14T06:44:01.2274088Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2274288Z collecting ... collected 1 item 2023-11-14T06:44:01.2275187Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2023-11-14T06:44:01.2275209Z 2023-11-14T06:44:01.2277121Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2277276Z PASSED [5.6110s] [100%] 2023-11-14T06:44:01.2277295Z 2023-11-14T06:44:01.2278532Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-846941ae5d294aea.xml - 2023-11-14T06:44:01.2278823Z ============================== 1 passed in 5.65s =============================== 2023-11-14T06:44:01.2279836Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4f564cba4b03de8.xml 2023-11-14T06:44:01.2280121Z ============================= test session starts ============================== 2023-11-14T06:44:01.2280762Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2280918Z cachedir: .pytest_cache 2023-11-14T06:44:01.2281855Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2282043Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2282196Z configfile: pytest.ini 2023-11-14T06:44:01.2283017Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2283195Z collecting ... collected 1 item 2023-11-14T06:44:01.2284056Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2023-11-14T06:44:01.2284076Z 2023-11-14T06:44:01.2285904Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2286568Z [1699944023.402107] [99f90b6e6ff3:11197:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2287275Z [1699944023.402107] [99f90b6e6ff3:11197:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2287926Z [1699944023.410339] [99f90b6e6ff3:11198:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2288623Z [1699944023.410339] [99f90b6e6ff3:11198:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2288775Z PASSED [6.4155s] [100%] 2023-11-14T06:44:01.2288795Z 2023-11-14T06:44:01.2290046Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4f564cba4b03de8.xml - 2023-11-14T06:44:01.2290328Z ============================== 1 passed in 6.45s =============================== 2023-11-14T06:44:01.2291600Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6b1c498128c165d.xml 2023-11-14T06:44:01.2291893Z ============================= test session starts ============================== 2023-11-14T06:44:01.2292591Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2292904Z cachedir: .pytest_cache 2023-11-14T06:44:01.2293778Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2293954Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2294107Z configfile: pytest.ini 2023-11-14T06:44:01.2294925Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2295094Z collecting ... collected 1 item 2023-11-14T06:44:01.2295969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2023-11-14T06:44:01.2295989Z 2023-11-14T06:44:01.2297851Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2298518Z [1699944035.258168] [99f90b6e6ff3:11224:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2299219Z [1699944035.258168] [99f90b6e6ff3:11224:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2299867Z [1699944035.265489] [99f90b6e6ff3:11225:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2300564Z [1699944035.265489] [99f90b6e6ff3:11225:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2300718Z PASSED [6.3117s] [100%] 2023-11-14T06:44:01.2300738Z 2023-11-14T06:44:01.2302138Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6b1c498128c165d.xml - 2023-11-14T06:44:01.2302444Z ============================== 1 passed in 6.35s =============================== 2023-11-14T06:44:01.2303460Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce7d64a59f351fc6.xml 2023-11-14T06:44:01.2303751Z ============================= test session starts ============================== 2023-11-14T06:44:01.2304393Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2304549Z cachedir: .pytest_cache 2023-11-14T06:44:01.2305407Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2305584Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2305739Z configfile: pytest.ini 2023-11-14T06:44:01.2306564Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2306733Z collecting ... collected 1 item 2023-11-14T06:44:01.2307659Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2023-11-14T06:44:01.2307680Z 2023-11-14T06:44:01.2309573Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2310231Z [1699944046.772711] [99f90b6e6ff3:11249:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2310933Z [1699944046.772711] [99f90b6e6ff3:11249:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2311586Z [1699944046.767790] [99f90b6e6ff3:11248:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2312402Z [1699944046.767790] [99f90b6e6ff3:11248:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2312556Z PASSED [6.4009s] [100%] 2023-11-14T06:44:01.2312576Z 2023-11-14T06:44:01.2313831Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce7d64a59f351fc6.xml - 2023-11-14T06:44:01.2314114Z ============================== 1 passed in 6.44s =============================== 2023-11-14T06:44:01.2315129Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4c59b48a3700c63.xml 2023-11-14T06:44:01.2315418Z ============================= test session starts ============================== 2023-11-14T06:44:01.2316071Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2316230Z cachedir: .pytest_cache 2023-11-14T06:44:01.2317096Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2317278Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2317432Z configfile: pytest.ini 2023-11-14T06:44:01.2318250Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2318418Z collecting ... collected 1 item 2023-11-14T06:44:01.2319188Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2023-11-14T06:44:01.2319226Z 2023-11-14T06:44:01.2321061Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2321742Z STAGE:2023-11-14 06:40:58 11276:11276 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2322413Z STAGE:2023-11-14 06:40:58 11275:11275 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2322999Z [1699944058.389931] [99f90b6e6ff3:11275:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2323598Z [1699944058.395846] [99f90b6e6ff3:11276:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2324252Z [1699944058.517231] [99f90b6e6ff3:11276:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2324982Z [1699944058.517231] [99f90b6e6ff3:11276:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2325660Z [1699944058.514565] [99f90b6e6ff3:11275:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2326336Z [1699944058.514565] [99f90b6e6ff3:11275:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2327768Z STAGE:2023-11-14 06:40:58 11276:11276 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:40:58 11275:11275 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2327789Z 2023-11-14T06:44:01.2328501Z STAGE:2023-11-14 06:40:58 11276:11276 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2329209Z STAGE:2023-11-14 06:40:58 11275:11275 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2329878Z STAGE:2023-11-14 06:40:59 11276:11276 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2330533Z STAGE:2023-11-14 06:40:59 11275:11275 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2331466Z STAGE:2023-11-14 06:40:59 11276:11276 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2332142Z STAGE:2023-11-14 06:40:59 11275:11275 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2332999Z STAGE:2023-11-14 06:40:59 11276:11276 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2333716Z STAGE:2023-11-14 06:40:59 11275:11275 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2333867Z PASSED [6.4000s] [100%] 2023-11-14T06:44:01.2333887Z 2023-11-14T06:44:01.2335128Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4c59b48a3700c63.xml - 2023-11-14T06:44:01.2335412Z ============================== 1 passed in 6.44s =============================== 2023-11-14T06:44:01.2336431Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce0c33e2dabc122b.xml 2023-11-14T06:44:01.2336720Z ============================= test session starts ============================== 2023-11-14T06:44:01.2337367Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2337522Z cachedir: .pytest_cache 2023-11-14T06:44:01.2338385Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2338562Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2338715Z configfile: pytest.ini 2023-11-14T06:44:01.2339533Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2339701Z collecting ... collected 1 item 2023-11-14T06:44:01.2340511Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2023-11-14T06:44:01.2340638Z 2023-11-14T06:44:01.2342446Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2343112Z STAGE:2023-11-14 06:41:10 11299:11299 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2343776Z STAGE:2023-11-14 06:41:10 11298:11298 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2344365Z [1699944070.230442] [99f90b6e6ff3:11298:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2344959Z [1699944070.230471] [99f90b6e6ff3:11299:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2345614Z [1699944070.356919] [99f90b6e6ff3:11299:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2346302Z [1699944070.356919] [99f90b6e6ff3:11299:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2347822Z STAGE:2023-11-14 06:41:10 11299:11299 ActivityProfilerController.cpp:320] Completed Stage: Collection[1699944070.356623] [99f90b6e6ff3:11298:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2348503Z [1699944070.356623] [99f90b6e6ff3:11298:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2348525Z 2023-11-14T06:44:01.2349222Z STAGE:2023-11-14 06:41:10 11298:11298 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2350689Z STAGE:2023-11-14 06:41:10 11299:11299 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:41:10 11298:11298 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2350710Z 2023-11-14T06:44:01.2351364Z STAGE:2023-11-14 06:41:10 11299:11299 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2352020Z STAGE:2023-11-14 06:41:10 11298:11298 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2352791Z STAGE:2023-11-14 06:41:10 11299:11299 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2353513Z STAGE:2023-11-14 06:41:10 11299:11299 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2354181Z STAGE:2023-11-14 06:41:10 11298:11298 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2354884Z STAGE:2023-11-14 06:41:10 11298:11298 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2355058Z PASSED [6.4018s] [100%] 2023-11-14T06:44:01.2355077Z 2023-11-14T06:44:01.2356328Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce0c33e2dabc122b.xml - 2023-11-14T06:44:01.2356607Z ============================== 1 passed in 6.44s =============================== 2023-11-14T06:44:01.2357620Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86d5827f6bbebbdc.xml 2023-11-14T06:44:01.2357893Z ============================= test session starts ============================== 2023-11-14T06:44:01.2358553Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2358711Z cachedir: .pytest_cache 2023-11-14T06:44:01.2359554Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2359747Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2359900Z configfile: pytest.ini 2023-11-14T06:44:01.2360714Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2360971Z collecting ... collected 1 item 2023-11-14T06:44:01.2361750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2023-11-14T06:44:01.2361777Z 2023-11-14T06:44:01.2363547Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2363848Z SKIPPED [5.5114s] (Skipped due to small world size.) [100%] 2023-11-14T06:44:01.2363867Z 2023-11-14T06:44:01.2365126Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86d5827f6bbebbdc.xml - 2023-11-14T06:44:01.2365391Z ============================== 1 skipped in 5.55s ============================== 2023-11-14T06:44:01.2366410Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-194bd096aeeecc7b.xml 2023-11-14T06:44:01.2366697Z ============================= test session starts ============================== 2023-11-14T06:44:01.2367349Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2367505Z cachedir: .pytest_cache 2023-11-14T06:44:01.2368370Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2368548Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2368721Z configfile: pytest.ini 2023-11-14T06:44:01.2369522Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2369691Z collecting ... collected 1 item 2023-11-14T06:44:01.2370399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2023-11-14T06:44:01.2370426Z 2023-11-14T06:44:01.2372338Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2373171Z STAGE:2023-11-14 06:41:32 11334:11334 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2373817Z STAGE:2023-11-14 06:41:32 11333:11333 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2374405Z [1699944092.721947] [99f90b6e6ff3:11333:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2374997Z [1699944092.727765] [99f90b6e6ff3:11334:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2375653Z [1699944092.842434] [99f90b6e6ff3:11333:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2376341Z [1699944092.842434] [99f90b6e6ff3:11333:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2377008Z [1699944092.844953] [99f90b6e6ff3:11334:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2377691Z [1699944092.844953] [99f90b6e6ff3:11334:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2379120Z STAGE:2023-11-14 06:41:33 11333:11333 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:41:33 11334:11334 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2379142Z 2023-11-14T06:44:01.2379847Z STAGE:2023-11-14 06:41:33 11333:11333 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2380569Z STAGE:2023-11-14 06:41:33 11334:11334 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2381331Z STAGE:2023-11-14 06:41:33 11334:11334 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2381994Z STAGE:2023-11-14 06:41:33 11333:11333 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2382685Z STAGE:2023-11-14 06:41:33 11334:11334 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2383341Z STAGE:2023-11-14 06:41:33 11333:11333 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2384043Z STAGE:2023-11-14 06:41:33 11334:11334 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2384760Z STAGE:2023-11-14 06:41:33 11333:11333 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2384912Z PASSED [6.3125s] [100%] 2023-11-14T06:44:01.2384932Z 2023-11-14T06:44:01.2386200Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-194bd096aeeecc7b.xml - 2023-11-14T06:44:01.2386465Z ============================== 1 passed in 6.35s =============================== 2023-11-14T06:44:01.2387480Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e5aa35540ffead0.xml 2023-11-14T06:44:01.2387775Z ============================= test session starts ============================== 2023-11-14T06:44:01.2388422Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2388579Z cachedir: .pytest_cache 2023-11-14T06:44:01.2389442Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2389619Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2389789Z configfile: pytest.ini 2023-11-14T06:44:01.2390588Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2390764Z collecting ... collected 1 item 2023-11-14T06:44:01.2391585Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2023-11-14T06:44:01.2391695Z 2023-11-14T06:44:01.2393756Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA reduce_scatter_tensor) [100%] 2023-11-14T06:44:01.2393778Z 2023-11-14T06:44:01.2395039Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e5aa35540ffead0.xml - 2023-11-14T06:44:01.2395306Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2396318Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0906156a0e59ab14.xml 2023-11-14T06:44:01.2396606Z ============================= test session starts ============================== 2023-11-14T06:44:01.2397256Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2397430Z cachedir: .pytest_cache 2023-11-14T06:44:01.2398272Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2398450Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2398623Z configfile: pytest.ini 2023-11-14T06:44:01.2399419Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2399588Z collecting ... collected 1 item 2023-11-14T06:44:01.2400329Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2023-11-14T06:44:01.2400349Z 2023-11-14T06:44:01.2402288Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA reduce) [100%] 2023-11-14T06:44:01.2402319Z 2023-11-14T06:44:01.2403594Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0906156a0e59ab14.xml - 2023-11-14T06:44:01.2403860Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2404873Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fb8a1365b69d744.xml 2023-11-14T06:44:01.2405162Z ============================= test session starts ============================== 2023-11-14T06:44:01.2405804Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2405984Z cachedir: .pytest_cache 2023-11-14T06:44:01.2406829Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2407012Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2407184Z configfile: pytest.ini 2023-11-14T06:44:01.2407984Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2408153Z collecting ... collected 1 item 2023-11-14T06:44:01.2408902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2023-11-14T06:44:01.2408923Z 2023-11-14T06:44:01.2410640Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2411560Z STAGE:2023-11-14 06:41:54 11363:11363 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2412205Z STAGE:2023-11-14 06:41:54 11362:11362 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2412935Z [1699944114.909852] [99f90b6e6ff3:11362:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2413533Z [1699944114.911901] [99f90b6e6ff3:11363:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2414192Z [1699944115.031938] [99f90b6e6ff3:11363:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2414890Z [1699944115.031938] [99f90b6e6ff3:11363:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2415532Z [1699944115.031217] [99f90b6e6ff3:11362:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2416217Z [1699944115.031217] [99f90b6e6ff3:11362:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2417642Z STAGE:2023-11-14 06:41:55 11363:11363 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:41:55 11362:11362 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2417670Z 2023-11-14T06:44:01.2418378Z STAGE:2023-11-14 06:41:55 11363:11363 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2419105Z STAGE:2023-11-14 06:41:55 11362:11362 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2419751Z STAGE:2023-11-14 06:41:55 11362:11362 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2420416Z STAGE:2023-11-14 06:41:55 11362:11362 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2421240Z STAGE:2023-11-14 06:41:55 11362:11362 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2421905Z STAGE:2023-11-14 06:41:55 11363:11363 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2422597Z STAGE:2023-11-14 06:41:55 11363:11363 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2423303Z STAGE:2023-11-14 06:41:55 11363:11363 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2423455Z PASSED [6.4018s] [100%] 2023-11-14T06:44:01.2423475Z 2023-11-14T06:44:01.2424742Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fb8a1365b69d744.xml - 2023-11-14T06:44:01.2425049Z ============================== 1 passed in 6.44s =============================== 2023-11-14T06:44:01.2426061Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b40bc8933902d6.xml 2023-11-14T06:44:01.2426353Z ============================= test session starts ============================== 2023-11-14T06:44:01.2426997Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2427180Z cachedir: .pytest_cache 2023-11-14T06:44:01.2428025Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2428201Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2428378Z configfile: pytest.ini 2023-11-14T06:44:01.2429176Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2429346Z collecting ... collected 1 item 2023-11-14T06:44:01.2430037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2023-11-14T06:44:01.2430057Z 2023-11-14T06:44:01.2431927Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (CPU tensor ops not supported by UCP TL) [100%] 2023-11-14T06:44:01.2432039Z 2023-11-14T06:44:01.2433320Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b40bc8933902d6.xml - 2023-11-14T06:44:01.2433585Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2434618Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d382e3a8df8128e.xml 2023-11-14T06:44:01.2434888Z ============================= test session starts ============================== 2023-11-14T06:44:01.2435535Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2435713Z cachedir: .pytest_cache 2023-11-14T06:44:01.2436560Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2436736Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2436916Z configfile: pytest.ini 2023-11-14T06:44:01.2437714Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2437883Z collecting ... collected 1 item 2023-11-14T06:44:01.2438657Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2023-11-14T06:44:01.2438678Z 2023-11-14T06:44:01.2440672Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires backend ucc to be one of {'gloo'}) [100%] 2023-11-14T06:44:01.2440693Z 2023-11-14T06:44:01.2442059Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d382e3a8df8128e.xml - 2023-11-14T06:44:01.2442347Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2443385Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9d13556bd5f71e8.xml 2023-11-14T06:44:01.2443655Z ============================= test session starts ============================== 2023-11-14T06:44:01.2444301Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2444477Z cachedir: .pytest_cache 2023-11-14T06:44:01.2445318Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2445494Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2445667Z configfile: pytest.ini 2023-11-14T06:44:01.2446469Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2446659Z collecting ... collected 1 item 2023-11-14T06:44:01.2447347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2023-11-14T06:44:01.2447368Z 2023-11-14T06:44:01.2449037Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2449642Z [1699944137.339058] [99f90b6e6ff3:11391:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2450219Z [1699944137.345297] [99f90b6e6ff3:11392:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2451125Z [1699944137.449314] [99f90b6e6ff3:11391:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2451818Z [1699944137.449314] [99f90b6e6ff3:11391:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2452658Z [1699944137.450538] [99f90b6e6ff3:11392:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2453354Z [1699944137.450538] [99f90b6e6ff3:11392:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2453507Z PASSED [6.0982s] [100%] 2023-11-14T06:44:01.2453528Z 2023-11-14T06:44:01.2454805Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9d13556bd5f71e8.xml - 2023-11-14T06:44:01.2455068Z ============================== 1 passed in 6.13s =============================== 2023-11-14T06:44:01.2456081Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76d9f49a1d6e9622.xml 2023-11-14T06:44:01.2456367Z ============================= test session starts ============================== 2023-11-14T06:44:01.2457023Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2457180Z cachedir: .pytest_cache 2023-11-14T06:44:01.2458059Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2458237Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2458412Z configfile: pytest.ini 2023-11-14T06:44:01.2459210Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2459381Z collecting ... collected 1 item 2023-11-14T06:44:01.2460379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2023-11-14T06:44:01.2460403Z 2023-11-14T06:44:01.2462546Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (ucc does not support send/recv from any source) [100%] 2023-11-14T06:44:01.2462575Z 2023-11-14T06:44:01.2463833Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76d9f49a1d6e9622.xml - 2023-11-14T06:44:01.2464099Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2465111Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b80533a64bca84a.xml 2023-11-14T06:44:01.2465398Z ============================= test session starts ============================== 2023-11-14T06:44:01.2466048Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2466209Z cachedir: .pytest_cache 2023-11-14T06:44:01.2467078Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2467259Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2467429Z configfile: pytest.ini 2023-11-14T06:44:01.2468228Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2468398Z collecting ... collected 1 item 2023-11-14T06:44:01.2469220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2023-11-14T06:44:01.2469241Z 2023-11-14T06:44:01.2471043Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2471720Z STAGE:2023-11-14 06:42:34 11413:11413 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2472477Z STAGE:2023-11-14 06:42:34 11414:11414 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2473065Z [1699944154.190052] [99f90b6e6ff3:11413:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2473661Z [1699944154.196098] [99f90b6e6ff3:11414:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2474319Z [1699944154.312712] [99f90b6e6ff3:11414:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2475018Z [1699944154.312712] [99f90b6e6ff3:11414:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2475704Z STAGE:2023-11-14 06:42:34 11414:11414 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2476357Z [1699944154.310590] [99f90b6e6ff3:11413:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2477060Z [1699944154.310590] [99f90b6e6ff3:11413:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2477739Z STAGE:2023-11-14 06:42:34 11413:11413 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2478444Z STAGE:2023-11-14 06:42:34 11414:11414 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2479164Z STAGE:2023-11-14 06:42:34 11413:11413 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2479317Z PASSED [6.4157s] [100%] 2023-11-14T06:44:01.2479338Z 2023-11-14T06:44:01.2480696Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b80533a64bca84a.xml - 2023-11-14T06:44:01.2480977Z ============================== 1 passed in 6.45s =============================== 2023-11-14T06:44:01.2481998Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c9785bf2006c466.xml 2023-11-14T06:44:01.2482288Z ============================= test session starts ============================== 2023-11-14T06:44:01.2482936Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2483108Z cachedir: .pytest_cache 2023-11-14T06:44:01.2483952Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2484129Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2484304Z configfile: pytest.ini 2023-11-14T06:44:01.2485107Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2485278Z collecting ... collected 1 item 2023-11-14T06:44:01.2486011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2023-11-14T06:44:01.2486038Z 2023-11-14T06:44:01.2487817Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (NCCL Send Recv Only) [100%] 2023-11-14T06:44:01.2487838Z 2023-11-14T06:44:01.2489092Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c9785bf2006c466.xml - 2023-11-14T06:44:01.2489361Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2490376Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90ac7a0c14864b98.xml 2023-11-14T06:44:01.2490865Z ============================= test session starts ============================== 2023-11-14T06:44:01.2491551Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2491860Z cachedir: .pytest_cache 2023-11-14T06:44:01.2492711Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2492887Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2493058Z configfile: pytest.ini 2023-11-14T06:44:01.2493856Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2494028Z collecting ... collected 1 item 2023-11-14T06:44:01.2494833Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2023-11-14T06:44:01.2494853Z 2023-11-14T06:44:01.2496630Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2497319Z STAGE:2023-11-14 06:42:51 11439:11439 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2497964Z STAGE:2023-11-14 06:42:51 11440:11440 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:44:01.2498550Z [1699944171.351232] [99f90b6e6ff3:11439:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2499144Z [1699944171.357574] [99f90b6e6ff3:11440:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2499800Z [1699944171.475010] [99f90b6e6ff3:11440:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2500710Z [1699944171.475010] [99f90b6e6ff3:11440:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2501420Z STAGE:2023-11-14 06:42:51 11440:11440 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2502082Z [1699944171.474309] [99f90b6e6ff3:11439:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2502780Z [1699944171.474309] [99f90b6e6ff3:11439:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2503457Z STAGE:2023-11-14 06:42:51 11439:11439 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:44:01.2504188Z STAGE:2023-11-14 06:42:51 11440:11440 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2504892Z STAGE:2023-11-14 06:42:51 11439:11439 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:44:01.2505042Z PASSED [6.6114s] [100%] 2023-11-14T06:44:01.2505069Z 2023-11-14T06:44:01.2506333Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90ac7a0c14864b98.xml - 2023-11-14T06:44:01.2506601Z ============================== 1 passed in 6.65s =============================== 2023-11-14T06:44:01.2507614Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-058c4e866f2e1040.xml 2023-11-14T06:44:01.2507902Z ============================= test session starts ============================== 2023-11-14T06:44:01.2508550Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2508725Z cachedir: .pytest_cache 2023-11-14T06:44:01.2509569Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2509745Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2509923Z configfile: pytest.ini 2023-11-14T06:44:01.2510725Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2510994Z collecting ... collected 1 item 2023-11-14T06:44:01.2511761Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2023-11-14T06:44:01.2511782Z 2023-11-14T06:44:01.2513515Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2514122Z [1699944183.325904] [99f90b6e6ff3:11462:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2514697Z [1699944183.332669] [99f90b6e6ff3:11463:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2515356Z [1699944183.440120] [99f90b6e6ff3:11463:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2516056Z [1699944183.440120] [99f90b6e6ff3:11463:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2516705Z [1699944183.438222] [99f90b6e6ff3:11462:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2517390Z [1699944183.438222] [99f90b6e6ff3:11462:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2517541Z PASSED [6.2102s] [100%] 2023-11-14T06:44:01.2517561Z 2023-11-14T06:44:01.2518810Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-058c4e866f2e1040.xml - 2023-11-14T06:44:01.2519095Z ============================== 1 passed in 6.25s =============================== 2023-11-14T06:44:01.2520204Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8547f02cc96e59c8.xml 2023-11-14T06:44:01.2520506Z ============================= test session starts ============================== 2023-11-14T06:44:01.2521164Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2521322Z cachedir: .pytest_cache 2023-11-14T06:44:01.2522184Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2522362Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2522516Z configfile: pytest.ini 2023-11-14T06:44:01.2523330Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2523502Z collecting ... collected 1 item 2023-11-14T06:44:01.2524286Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2023-11-14T06:44:01.2524307Z 2023-11-14T06:44:01.2526105Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2526701Z [1699944194.774250] [99f90b6e6ff3:11481:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2527297Z [1699944194.780894] [99f90b6e6ff3:11482:0] ucc_context.c:402 UCC ERROR failed to create tl context for cuda 2023-11-14T06:44:01.2527950Z [1699944194.891220] [99f90b6e6ff3:11482:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2528651Z [1699944194.891220] [99f90b6e6ff3:11482:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2529299Z [1699944194.885789] [99f90b6e6ff3:11481:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2529978Z [1699944194.885789] [99f90b6e6ff3:11481:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2530250Z PASSED [6.1143s] [100%] 2023-11-14T06:44:01.2530270Z 2023-11-14T06:44:01.2531839Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8547f02cc96e59c8.xml - 2023-11-14T06:44:01.2532129Z ============================== 1 passed in 6.15s =============================== 2023-11-14T06:44:01.2533151Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e362f7bbb37cd63.xml 2023-11-14T06:44:01.2533420Z ============================= test session starts ============================== 2023-11-14T06:44:01.2534084Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2534241Z cachedir: .pytest_cache 2023-11-14T06:44:01.2535083Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2535284Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2535438Z configfile: pytest.ini 2023-11-14T06:44:01.2536257Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2536431Z collecting ... collected 1 item 2023-11-14T06:44:01.2537229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2023-11-14T06:44:01.2537250Z 2023-11-14T06:44:01.2539507Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl & Gloo backend support DistributedDataParallel) [100%] 2023-11-14T06:44:01.2539532Z 2023-11-14T06:44:01.2540801Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e362f7bbb37cd63.xml - 2023-11-14T06:44:01.2541091Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2542097Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2698b952ff59b0bd.xml 2023-11-14T06:44:01.2542365Z ============================= test session starts ============================== 2023-11-14T06:44:01.2543028Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2543184Z cachedir: .pytest_cache 2023-11-14T06:44:01.2544033Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2544230Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2544384Z configfile: pytest.ini 2023-11-14T06:44:01.2545204Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2545379Z collecting ... collected 1 item 2023-11-14T06:44:01.2546097Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2023-11-14T06:44:01.2546117Z 2023-11-14T06:44:01.2548176Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl & Gloo backend support DistributedDataParallel) [100%] 2023-11-14T06:44:01.2548197Z 2023-11-14T06:44:01.2549442Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2698b952ff59b0bd.xml - 2023-11-14T06:44:01.2549726Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:44:01.2550866Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb864753fae87bf2.xml 2023-11-14T06:44:01.2551136Z ============================= test session starts ============================== 2023-11-14T06:44:01.2551798Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2551954Z cachedir: .pytest_cache 2023-11-14T06:44:01.2552815Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2552995Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2553151Z configfile: pytest.ini 2023-11-14T06:44:01.2553972Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2554143Z collecting ... collected 1 item 2023-11-14T06:44:01.2555017Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2023-11-14T06:44:01.2555044Z 2023-11-14T06:44:01.2558205Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2558895Z [1699944216.918670] [99f90b6e6ff3:11510:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2559597Z [1699944216.918670] [99f90b6e6ff3:11510:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2563073Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:44:01.2563787Z [1699944216.924575] [99f90b6e6ff3:11511:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2564472Z [1699944216.924575] [99f90b6e6ff3:11511:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2567883Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:44:01.2568048Z PASSED [6.2027s] [100%] 2023-11-14T06:44:01.2568068Z 2023-11-14T06:44:01.2569350Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb864753fae87bf2.xml - 2023-11-14T06:44:01.2569631Z ============================== 1 passed in 6.24s =============================== 2023-11-14T06:44:01.2570652Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d3ca16e27fede0d.xml 2023-11-14T06:44:01.2571199Z ============================= test session starts ============================== 2023-11-14T06:44:01.2571865Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2572024Z cachedir: .pytest_cache 2023-11-14T06:44:01.2572886Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2573214Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2573367Z configfile: pytest.ini 2023-11-14T06:44:01.2574192Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2574363Z collecting ... collected 1 item 2023-11-14T06:44:01.2575242Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2023-11-14T06:44:01.2575262Z 2023-11-14T06:44:01.2577113Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2577777Z [1699944228.537949] [99f90b6e6ff3:11538:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2578489Z [1699944228.537949] [99f90b6e6ff3:11538:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2579343Z [1699944233.982307] [99f90b6e6ff3:11538:0] tag_match.c:62 UCX WARN unexpected tag-receive descriptor 0xb336340 was not matched 2023-11-14T06:44:01.2580014Z [1699944228.537353] [99f90b6e6ff3:11537:0] parser.c:2034 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-11-14T06:44:01.2580692Z [1699944228.537353] [99f90b6e6ff3:11537:0] parser.c:2034 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-11-14T06:44:01.2581825Z [1699944233.950799] [99f90b6e6ff3:11537:1] ucc_schedule.h:192 UCC WARN timeout 5 sec. has expired on Barrier; CL_BASIC {TL_UCP}, team_id 1 rank 0, ctx_rank 0, seq_num 5, req 0x880b180 2023-11-14T06:44:01.2582764Z [1699944233.972324] [99f90b6e6ff3:11537:0] mpool.c:54 UCX WARN object 0x89156c0 {{cb|rcv_tag} recv length 0 host memory} was not returned to mpool ucp_requests 2023-11-14T06:44:01.2582926Z PASSED [11.2072s] [100%] 2023-11-14T06:44:01.2582946Z 2023-11-14T06:44:01.2584234Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d3ca16e27fede0d.xml - 2023-11-14T06:44:01.2584497Z ============================== 1 passed in 11.24s ============================== 2023-11-14T06:44:01.2585507Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae9633216f6f3e2d.xml 2023-11-14T06:44:01.2585796Z ============================= test session starts ============================== 2023-11-14T06:44:01.2586450Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:44:01.2586624Z cachedir: .pytest_cache 2023-11-14T06:44:01.2587470Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:44:01.2587653Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:44:01.2587824Z configfile: pytest.ini 2023-11-14T06:44:01.2588624Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:44:01.2588792Z collecting ... collected 1 item 2023-11-14T06:44:01.2589684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2023-11-14T06:44:01.2589704Z 2023-11-14T06:44:01.2591576Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:44:01.2592117Z SKIPPED [0.2850s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [100%] 2023-11-14T06:44:01.2592237Z 2023-11-14T06:44:01.2593502Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae9633216f6f3e2d.xml - 2023-11-14T06:44:01.2593767Z ============================== 1 skipped in 0.32s ============================== 2023-11-14T06:44:01.2593965Z Traceback (most recent call last): 2023-11-14T06:44:01.2594590Z File "/var/lib/jenkins/workspace/test/distributed/test_distributed_spawn.py", line 59, in 2023-11-14T06:44:01.2594748Z run_tests() 2023-11-14T06:44:01.2595633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1011, in run_tests 2023-11-14T06:44:01.2595995Z assert len(failed_tests) == 0, "{} unit test(s) failed:\n\t{}".format( 2023-11-14T06:44:01.2596199Z AssertionError: 2 unit test(s) failed: 2023-11-14T06:44:01.2596726Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2023-11-14T06:44:01.2597301Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2023-11-14T06:44:01.2597339Z 2023-11-14T06:44:01.2598623Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn 1/2 (/var/lib/jenkins/workspace/test/test-reports/distributed.test_distributed_spawn_1.2_o9c4kvs8_toprint.log) 2023-11-14T06:44:01.2598644Z 2023-11-14T06:44:01.2598870Z distributed/test_distributed_spawn 1/2 failed! 2023-11-14T06:44:01.2599473Z Starting test batch 'probable_relevance' 6520.401940345764 seconds after initiating testing 2023-11-14T06:44:01.2599683Z With sharding, this batch will run 16 tests 2023-11-14T06:44:03.0113345Z Running distributed/fsdp/test_fsdp_optim_state 1/1 ... [2023-11-14 06:44:03.010763] 2023-11-14T06:44:03.0116663Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:44:03.011094] 2023-11-14T06:49:41.6337858Z 2023-11-14T06:49:41.6344015Z distributed/fsdp/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_8222374d7fd53e54_.log 2023-11-14T06:49:41.6409070Z Running 60 items in this shard: test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_compatible_with_trec, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_nested, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_keys, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_nested_invalid, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_interface_arguments, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_no_grad, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_input_warning, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_without_param_groups, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_names, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type_StateDictType_FULL_STATE_DICT, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type_StateDictType_SHARDED_STATE_DICT, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_state_dict_with_none_tensor_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_use_orig_params, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_empty_optimizer_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_no_shard 2023-11-14T06:49:41.6470784Z 2023-11-14T06:49:41.6471368Z Running distributed/rpc/test_tensorpipe_agent 1/1 ... [2023-11-14 06:49:41.636575] 2023-11-14T06:49:41.6473912Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '--shard-id=0', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:49:41.636838] 2023-11-14T06:49:47.3270192Z 2023-11-14T06:49:47.3272244Z distributed/rpc/test_tensorpipe_agent 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_tensorpipe_agent_1.1_ea1ae83cfd92a020_.log 2023-11-14T06:49:47.3274094Z 2023-11-14T06:49:47.3274925Z Running distributed/test_multi_threaded_pg 1/1 ... [2023-11-14 06:49:47.327176] 2023-11-14T06:49:47.3279072Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:49:47.327441] 2023-11-14T06:49:50.7488084Z 2023-11-14T06:49:50.7490068Z distributed/test_multi_threaded_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_multi_threaded_pg_1.1_635ac9df2e2dbdde_.log 2023-11-14T06:49:50.7504458Z Running 19 items in this shard: test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_skip, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_coalesced, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_ops, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_to_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_allgather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_assert_equal_on_rank, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_bwd_sees_fwd_pg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_gather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_reduce_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_subpg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_using_pg_from_another_thread 2023-11-14T06:49:50.7516720Z 2023-11-14T06:49:50.7517227Z Running distributed/test_c10d_nccl 1/1 ... [2023-11-14 06:49:50.749016] 2023-11-14T06:49:50.7519605Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=0', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:49:50.749329] 2023-11-14T07:06:15.8593520Z 2023-11-14T07:06:15.8595585Z distributed/test_c10d_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.1_3be3173cab9a1fc5_.log 2023-11-14T07:06:15.8600115Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2023-11-14T07:06:15.8602825Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2023-11-14T07:06:15.8604491Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2023-11-14T07:06:15.8605945Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_abort_pg 2023-11-14T07:06:15.8607971Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allgather_base_basics 2023-11-14T07:06:15.8612362Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allgather_base_ops 2023-11-14T07:06:15.8615188Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allgather_ops 2023-11-14T07:06:15.8617779Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allreduce_in_cudagraph 2023-11-14T07:06:15.8619285Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allreduce_ops 2023-11-14T07:06:15.8621011Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_alltoall_ops_with_cudafree_race 2023-11-14T07:06:15.8623808Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_barrier 2023-11-14T07:06:15.8627522Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_broadcast_ops 2023-11-14T07:06:15.8630785Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_close_pg 2023-11-14T07:06:15.8633786Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_empty_tensors 2023-11-14T07:06:15.8635210Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_gather_checks 2023-11-14T07:06:15.8636613Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_gather_ops 2023-11-14T07:06:15.8638022Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_gather_stress 2023-11-14T07:06:15.8640373Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_init_process_group_nccl_timeout 2023-11-14T07:06:15.8642710Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_nccl_dist_backend_error 2023-11-14T07:06:15.8644406Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_nccl_watchdog_cudagraph 2023-11-14T07:06:15.8645889Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_ops 2023-11-14T07:06:15.8647361Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_scatter_base_basics 2023-11-14T07:06:15.8648924Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_scatter_base_ops 2023-11-14T07:06:15.8650436Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_scatter_ops 2023-11-14T07:06:15.8652466Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_scatter_checks 2023-11-14T07:06:15.8653898Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_scatter_ops 2023-11-14T07:06:15.8655319Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_scatter_stress 2023-11-14T07:06:15.8656730Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_send_recv 2023-11-14T07:06:15.8658413Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_sparse_allreduce_ops 2023-11-14T07:06:15.8659929Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_tensor_register_hook 2023-11-14T07:06:15.8661551Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2023-11-14T07:06:15.8663382Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2023-11-14T07:06:15.8665216Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2023-11-14T07:06:15.8667052Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2023-11-14T07:06:15.8668928Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2023-11-14T07:06:15.8670649Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2023-11-14T07:06:15.8672342Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2023-11-14T07:06:15.8674136Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2023-11-14T07:06:15.8675880Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2023-11-14T07:06:15.8677594Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2023-11-14T07:06:15.8679284Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2023-11-14T07:06:15.8681061Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2023-11-14T07:06:15.8682894Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2023-11-14T07:06:15.8684761Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2023-11-14T07:06:15.8686651Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2023-11-14T07:06:15.8688624Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2023-11-14T07:06:15.8691099Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2023-11-14T07:06:15.8693191Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2023-11-14T07:06:15.8695102Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2023-11-14T07:06:15.8696975Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2023-11-14T07:06:15.8698906Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2023-11-14T07:06:15.8700901Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2023-11-14T07:06:15.8703080Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2023-11-14T07:06:15.8705092Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2023-11-14T07:06:15.8706986Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2023-11-14T07:06:15.8708826Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2023-11-14T07:06:15.8710770Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2023-11-14T07:06:15.8712680Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2023-11-14T07:06:15.8714562Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2023-11-14T07:06:15.8716344Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2023-11-14T07:06:15.8718029Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2023-11-14T07:06:15.8719630Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2023-11-14T07:06:15.8721261Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2023-11-14T07:06:15.8723053Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2023-11-14T07:06:15.8724844Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2023-11-14T07:06:15.8726541Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2023-11-14T07:06:15.8728271Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2023-11-14T07:06:15.8730125Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2023-11-14T07:06:15.8732548Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2023-11-14T07:06:15.8734500Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2023-11-14T07:06:15.8736526Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2023-11-14T07:06:15.8738526Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2023-11-14T07:06:15.8740229Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2023-11-14T07:06:15.8741808Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2023-11-14T07:06:15.8743533Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2023-11-14T07:06:15.8746245Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2023-11-14T07:06:15.8749556Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2023-11-14T07:06:15.8752818Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2023-11-14T07:06:15.8754494Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2023-11-14T07:06:15.8756237Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2023-11-14T07:06:15.8758113Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2023-11-14T07:06:15.8760086Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2023-11-14T07:06:15.8762056Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2023-11-14T07:06:15.8763976Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2023-11-14T07:06:15.8765636Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2023-11-14T07:06:15.8767407Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2023-11-14T07:06:15.8769391Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2023-11-14T07:06:15.8772063Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2023-11-14T07:06:15.8774090Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2023-11-14T07:06:15.8775956Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2023-11-14T07:06:15.8777542Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2023-11-14T07:06:15.8779111Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2023-11-14T07:06:15.8780749Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2023-11-14T07:06:15.8782384Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2023-11-14T07:06:15.8784180Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2023-11-14T07:06:15.8785954Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2023-11-14T07:06:15.8787703Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2023-11-14T07:06:15.8789389Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2023-11-14T07:06:15.8790937Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2023-11-14T07:06:15.8792418Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2023-11-14T07:06:15.8793903Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2023-11-14T07:06:15.8795484Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2023-11-14T07:06:15.8797293Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2023-11-14T07:06:15.8798911Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2023-11-14T07:06:15.8800550Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit 2023-11-14T07:06:15.8802226Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2023-11-14T07:06:15.8803873Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill 2023-11-14T07:06:15.8805514Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2023-11-14T07:06:15.8807110Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2023-11-14T07:06:15.8808611Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2023-11-14T07:06:15.8810007Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2023-11-14T07:06:15.8811936Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2023-11-14T07:06:15.8813257Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2023-11-14T07:06:15.8814564Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2023-11-14T07:06:15.8816175Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2023-11-14T07:06:15.8817671Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_timeout 2023-11-14T07:06:15.8819109Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2023-11-14T07:06:15.8820613Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2023-11-14T07:06:15.8822099Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2023-11-14T07:06:15.8823517Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2023-11-14T07:06:15.8824892Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2023-11-14T07:06:15.8826356Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2023-11-14T07:06:15.8827928Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2023-11-14T07:06:15.8829489Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2023-11-14T07:06:15.8830996Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2023-11-14T07:06:15.8832476Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2023-11-14T07:06:15.8833882Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2023-11-14T07:06:15.8835233Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2023-11-14T07:06:15.8836690Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_into_tensor_work_wait_gpu 2023-11-14T07:06:15.8838202Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_work_wait_gpu 2023-11-14T07:06:15.8839643Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_allreduce_work_wait_gpu 2023-11-14T07:06:15.8841229Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_alltoall_work_wait_gpu 2023-11-14T07:06:15.8842639Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_broadcast_work_wait_gpu 2023-11-14T07:06:15.8844118Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_consecutive_comm_work_wait_gpu 2023-11-14T07:06:15.8845614Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_nested_comm_tensor_wrapping 2023-11-14T07:06:15.8847040Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_base_k 2023-11-14T07:06:15.8848523Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_coalesced 2023-11-14T07:06:15.8850087Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_work_wait_gpu 2023-11-14T07:06:15.8852174Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_work_wait_gpu 2023-11-14T07:06:15.8853612Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_scatter_work_wait_gpu 2023-11-14T07:06:15.8855271Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2023-11-14T07:06:15.8857164Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2023-11-14T07:06:15.8859069Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2023-11-14T07:06:15.8861068Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2023-11-14T07:06:15.8863074Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2023-11-14T07:06:15.8865211Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2023-11-14T07:06:15.8866976Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2023-11-14T07:06:15.8868520Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2023-11-14T07:06:15.8870097Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2023-11-14T07:06:15.8871630Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2023-11-14T07:06:15.8872965Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2023-11-14T07:06:15.8874193Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short 2023-11-14T07:06:15.8875495Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active 2023-11-14T07:06:15.8876874Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck 2023-11-14T07:06:15.8877665Z 2023-11-14T07:06:15.8878317Z Running distributed/rpc/cuda/test_tensorpipe_agent 2/2 ... [2023-11-14 07:06:15.863663] 2023-11-14T07:06:15.8880938Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:06:15.863934] 2023-11-14T07:19:51.9989058Z 2023-11-14T07:19:51.9994098Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_2.2_bb2240a166d6047a_.log 2023-11-14T07:19:51.9998090Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device 2023-11-14T07:19:52.0001182Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device_script 2023-11-14T07:19:52.0003578Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_always 2023-11-14T07:19:52.0005528Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_except_last 2023-11-14T07:19:52.0007466Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never 2023-11-14T07:19:52.0009346Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_always 2023-11-14T07:19:52.0012377Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_nested_with_cuda_future 2023-11-14T07:19:52.0014938Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_callback_changes_devices 2023-11-14T07:19:52.0017214Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_sparse_tensor 2023-11-14T07:19:52.0019840Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor 2023-11-14T07:19:52.0022358Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_tensor 2023-11-14T07:19:52.0024647Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_device 2023-11-14T07:19:52.0026764Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_int 2023-11-14T07:19:52.0028840Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_str 2023-11-14T07:19:52.0030977Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_modify_tensor_inplace 2023-11-14T07:19:52.0033135Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_replace_tensor 2023-11-14T07:19:52.0035291Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_value_on_bad_device 2023-11-14T07:19:52.0037354Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream 2023-11-14T07:19:52.0039328Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_multi 2023-11-14T07:19:52.0041398Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested_multi 2023-11-14T07:19:52.0043528Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_default 2023-11-14T07:19:52.0045688Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_non_default 2023-11-14T07:19:52.0047971Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_7 2023-11-14T07:19:52.0050032Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_8 2023-11-14T07:19:52.0052634Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_1 2023-11-14T07:19:52.0054760Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_7 2023-11-14T07:19:52.0056906Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default 2023-11-14T07:19:52.0059029Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_default 2023-11-14T07:19:52.0061105Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_gpu 2023-11-14T07:19:52.0063208Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_local_device 2023-11-14T07:19:52.0065423Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_remote_device 2023-11-14T07:19:52.0067575Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_many_to_one 2023-11-14T07:19:52.0069818Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config 2023-11-14T07:19:52.0071998Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_loop 2023-11-14T07:19:52.0074184Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_not_timeout 2023-11-14T07:19:52.0076452Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote_response 2023-11-14T07:19:52.0078707Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response 2023-11-14T07:19:52.0080842Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu 2023-11-14T07:19:52.0082903Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu_self 2023-11-14T07:19:52.0084970Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_remote 2023-11-14T07:19:52.0087016Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch 2023-11-14T07:19:52.0089167Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization2 2023-11-14T07:19:52.0091909Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization4 2023-11-14T07:19:52.0094112Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization1 2023-11-14T07:19:52.0096427Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization2 2023-11-14T07:19:52.0098556Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization3 2023-11-14T07:19:52.0100667Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization4 2023-11-14T07:19:52.0102796Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization5 2023-11-14T07:19:52.0104942Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization3 2023-11-14T07:19:52.0107088Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization4 2023-11-14T07:19:52.0109219Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization1 2023-11-14T07:19:52.0111359Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization2 2023-11-14T07:19:52.0113494Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization4 2023-11-14T07:19:52.0115664Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_with_unpickleable_attributes 2023-11-14T07:19:52.0117907Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_tensor_view_as_return_value 2023-11-14T07:19:52.0120071Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_device_maps_backward_pass 2023-11-14T07:19:52.0121270Z 2023-11-14T07:19:52.0121891Z Running distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 ... [2023-11-14 07:19:52.000800] 2023-11-14T07:19:52.0124444Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_hsdp_dtensor_state_dict.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:19:52.001144] 2023-11-14T07:20:26.0313544Z 2023-11-14T07:20:26.0317861Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_5d62ceeca88f7d07_.log 2023-11-14T07:20:26.0326879Z Running 8 items in this shard: test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_hsdp_init_with_device_mesh, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_raises_tp_hsdp_not_supported_error 2023-11-14T07:20:26.0335761Z 2023-11-14T07:20:26.0336342Z Running distributed/test_c10d_object_collectives 1/1 ... [2023-11-14 07:20:26.031587] 2023-11-14T07:20:26.0338839Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:20:26.031890] 2023-11-14T07:20:59.6626629Z 2023-11-14T07:20:59.6630792Z distributed/test_c10d_object_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_object_collectives_1.1_1d83110884a90e03_.log 2023-11-14T07:20:59.6637439Z Running 8 items in this shard: test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_broadcast_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_scatter_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_broadcast_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_scatter_object 2023-11-14T07:20:59.6643003Z 2023-11-14T07:20:59.6643549Z Running distributed/fsdp/test_fsdp_grad_acc 1/1 ... [2023-11-14 07:20:59.662790] 2023-11-14T07:20:59.6646317Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:20:59.663055] 2023-11-14T07:21:49.4707450Z 2023-11-14T07:21:49.4709307Z distributed/fsdp/test_fsdp_grad_acc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_cb25fe4204c1ffbd_.log 2023-11-14T07:21:49.4718329Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_True 2023-11-14T07:21:49.4725317Z 2023-11-14T07:21:49.4726075Z Running distributed/_composable/fully_shard/test_fully_shard_compile 1/1 ... [2023-11-14 07:21:49.471066] 2023-11-14T07:21:49.4728858Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_compile.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:21:49.471330] 2023-11-14T07:22:36.5840431Z 2023-11-14T07:22:36.5842869Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_compile_1.1_928f8a9a29d1b5a7_.log 2023-11-14T07:22:36.5846060Z Running 1 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_compile.py::TestCompile::test_compile 2023-11-14T07:22:36.5847461Z 2023-11-14T07:22:36.5848114Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2023-11-14 07:22:36.584266] 2023-11-14T07:22:36.5851307Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:22:36.584531] 2023-11-14T07:24:14.8565365Z 2023-11-14T07:24:14.8571674Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_f9624234f34a6c61_.log 2023-11-14T07:24:14.8582702Z Running 18 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum 2023-11-14T07:24:14.8592888Z 2023-11-14T07:24:14.8593488Z Running distributed/fsdp/test_fsdp_flatten_params 1/1 ... [2023-11-14 07:24:14.856739] 2023-11-14T07:24:14.8596013Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:24:14.857019] 2023-11-14T07:25:14.0313292Z 2023-11-14T07:25:14.0315371Z distributed/fsdp/test_fsdp_flatten_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_f60b7b2b4b418f45_.log 2023-11-14T07:25:14.0327293Z Running 11 items in this shard: test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_empty_module, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_full_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_mixed_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_unaligned, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flatten_nothing, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_partial_flattening, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_pnorm_after_step_with_shared_params 2023-11-14T07:25:14.0336131Z 2023-11-14T07:25:14.0336735Z Running distributed/fsdp/test_fsdp_exec_order 1/1 ... [2023-11-14 07:25:14.031550] 2023-11-14T07:25:14.0339196Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:25:14.031840] 2023-11-14T07:25:57.6808094Z 2023-11-14T07:25:57.6812264Z distributed/fsdp/test_fsdp_exec_order 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_cd6c8a3c178e18a3_.log 2023-11-14T07:25:57.6821639Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP 2023-11-14T07:25:57.6829811Z 2023-11-14T07:25:57.6830419Z Running distributed/fsdp/test_checkpoint_wrapper 1/1 ... [2023-11-14 07:25:57.681030] 2023-11-14T07:25:57.6832911Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:25:57.681334] 2023-11-14T07:26:02.6551940Z 2023-11-14T07:26:02.6554885Z distributed/fsdp/test_checkpoint_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_2e1d514b518b636b_.log 2023-11-14T07:26:02.6563237Z Running 8 items in this shard: test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_apply_activation_checkpointing, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_args_kwargs, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_cpu_offload, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_kwarg_support, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_parity, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_forward_missing_attributes, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_fqn, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_load_activation_checkpointed_module 2023-11-14T07:26:02.6569010Z 2023-11-14T07:26:02.6569888Z Running distributed/test_data_parallel 1/1 ... [2023-11-14 07:26:02.655451] 2023-11-14T07:26:02.6572876Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_data_parallel.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:26:02.655736] 2023-11-14T07:26:08.0813434Z 2023-11-14T07:26:08.0815655Z distributed/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_data_parallel_1.1_82ef56b1e5dfddb7_.log 2023-11-14T07:26:08.0846619Z Running 46 items in this shard: test/distributed/test_data_parallel.py::TestDataParallel::test_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_buffers_requiring_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_complex, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_device_args, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_function_deletion, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_lazy_linear, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_device, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_no_refcycles, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_module_zero_inputs, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_multiple_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_output, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_no_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_rnn, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_small_back, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_sparse, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_different_len_dicts, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_passes_exception, test/distributed/test_data_parallel.py::TestDataParallel::test_parameter_list_dict_replica, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate_buffers, test/distributed/test_data_parallel.py::TestDataParallel::test_save_replica_module, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_strided_grad_layout, test/distributed/test_data_parallel.py::TestDataParallel::test_zero_grad, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 2023-11-14T07:26:08.0876037Z 2023-11-14T07:26:08.0876576Z Running distributed/_spmd/test_tracing 1/1 ... [2023-11-14 07:26:08.081628] 2023-11-14T07:26:08.0878933Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_spmd/test_tracing.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:26:08.081906] 2023-11-14T07:26:10.9002416Z 2023-11-14T07:26:10.9004795Z distributed/_spmd/test_tracing 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_tracing_1.1_0bb221544268a705_.log 2023-11-14T07:26:10.9006121Z 2023-11-14T07:26:10.9008269Z Running distributed/fsdp/test_fsdp_mixed_precision 1/1 ... [2023-11-14 07:26:10.900478] 2023-11-14T07:26:10.9012161Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:26:10.900761] 2023-11-14T07:32:28.1277958Z 2023-11-14T07:32:28.1282248Z distributed/fsdp/test_fsdp_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_abc742475dcd9b44_.log 2023-11-14T07:32:28.1346346Z Running 65 items in this shard: test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_eval_root_cast_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_buffers, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_comm, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_input_grads_with_param_mixed_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_resnet, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_False, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_True, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_default, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_only_params_and_bufs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_params_and_reduce_diff, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_reduce, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_e2e_full_shard, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionIgnoredModules::test_mixed_precision_with_ignored_module, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_external_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPTrainEval::test_train_ema_eval_flow 2023-11-14T07:32:28.1408245Z 2023-11-14T07:32:28.1408898Z Starting test batch 'unranked_relevance' 9427.654235124588 seconds after initiating testing 2023-11-14T07:32:28.1409818Z With sharding, this batch will run 42 tests 2023-11-14T07:32:30.2157938Z Running distributed/optim/test_zero_redundancy_optimizer 1/1 ... [2023-11-14 07:32:30.215191] 2023-11-14T07:32:30.2160620Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:32:30.215523] 2023-11-14T07:36:42.5091592Z 2023-11-14T07:36:42.5097494Z distributed/optim/test_zero_redundancy_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_9ddfd62101668295_.log 2023-11-14T07:36:42.5148931Z Running 42 items in this shard: test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_constructor, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_same_dense_param_type, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_state_dict, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_extra_inner_key, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_kwargs, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_without_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_zero_grad, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_add_param_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_collect_shards, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_multiple_param_groups, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_nondefault_process_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_sharding, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step_with_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_cpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_gpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_True 2023-11-14T07:36:42.5196484Z 2023-11-14T07:36:42.5197120Z Running distributed/fsdp/test_fsdp_checkpoint 1/1 ... [2023-11-14 07:36:42.509882] 2023-11-14T07:36:42.5199585Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:36:42.510228] 2023-11-14T07:38:16.8563981Z 2023-11-14T07:38:16.8568521Z distributed/fsdp/test_fsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_9c3167609316da8e_.log 2023-11-14T07:38:16.8590704Z Running 17 items in this shard: test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmodule::test_checkpoint_submodule_use_reentrant_False 2023-11-14T07:38:16.8611008Z 2023-11-14T07:38:16.8611488Z Running distributed/test_store 1/1 ... [2023-11-14 07:38:16.856547] 2023-11-14T07:38:16.8613814Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_store.py', '--shard-id=0', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:38:16.856808] 2023-11-14T07:44:53.5197074Z 2023-11-14T07:44:53.5199457Z distributed/test_store 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_store_1.1_1dec0fb1c10742a9_.log 2023-11-14T07:44:53.5202160Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append 2023-11-14T07:44:53.5203805Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_compare_set 2023-11-14T07:44:53.5205318Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file 2023-11-14T07:44:53.5206793Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_get 2023-11-14T07:44:53.5208644Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_set 2023-11-14T07:44:53.5210336Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_refcount 2023-11-14T07:44:53.5212677Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_set_get 2023-11-14T07:44:53.5214358Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_simple_wait 2023-11-14T07:44:53.5216141Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_append 2023-11-14T07:44:53.5217854Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_compare_set 2023-11-14T07:44:53.5219124Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_get 2023-11-14T07:44:53.5220348Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_set 2023-11-14T07:44:53.5221905Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_set_get 2023-11-14T07:44:53.5223153Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_simple_wait 2023-11-14T07:44:53.5224801Z Running 1 items in this shard: test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store 2023-11-14T07:44:53.5226159Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_append 2023-11-14T07:44:53.5227479Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set 2023-11-14T07:44:53.5228793Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get 2023-11-14T07:44:53.5230126Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set 2023-11-14T07:44:53.5231434Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_set_get 2023-11-14T07:44:53.5232762Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait 2023-11-14T07:44:53.5234109Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use 2023-11-14T07:44:53.5235383Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_append 2023-11-14T07:44:53.5236594Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_compare_set 2023-11-14T07:44:53.5237943Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket 2023-11-14T07:44:53.5239299Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_get 2023-11-14T07:44:53.5240518Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_set 2023-11-14T07:44:53.5241890Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size 2023-11-14T07:44:53.5243655Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size 2023-11-14T07:44:53.5245063Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multitenancy 2023-11-14T07:44:53.5246333Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys 2023-11-14T07:44:53.5247549Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_set_get 2023-11-14T07:44:53.5248762Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_simple_wait 2023-11-14T07:44:53.5250140Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients 2023-11-14T07:44:53.5252169Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket 2023-11-14T07:44:53.5253594Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use 2023-11-14T07:44:53.5254944Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_append 2023-11-14T07:44:53.5256234Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set 2023-11-14T07:44:53.5257662Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket 2023-11-14T07:44:53.5259091Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get 2023-11-14T07:44:53.5260375Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set 2023-11-14T07:44:53.5261967Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size 2023-11-14T07:44:53.5263594Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size 2023-11-14T07:44:53.5265082Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy 2023-11-14T07:44:53.5266432Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys 2023-11-14T07:44:53.5267742Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get 2023-11-14T07:44:53.5269006Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait 2023-11-14T07:44:53.5270447Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients 2023-11-14T07:44:53.5271966Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket 2023-11-14T07:44:53.5273313Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_append 2023-11-14T07:44:53.5274616Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set 2023-11-14T07:44:53.5275941Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get 2023-11-14T07:44:53.5277239Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set 2023-11-14T07:44:53.5278509Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get 2023-11-14T07:44:53.5279812Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait 2023-11-14T07:44:53.5281088Z Running 1 items in this shard: test/distributed/test_store.py::PythonStoreTest::test_set_get 2023-11-14T07:44:53.5282345Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_unknown_handler 2023-11-14T07:44:53.5283700Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_url_with_node_params 2023-11-14T07:44:53.5285016Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousEnvTest::test_nominal 2023-11-14T07:44:53.5286458Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_common_errors 2023-11-14T07:44:53.5287754Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_nominal 2023-11-14T07:44:53.5289063Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_common_errors 2023-11-14T07:44:53.5290388Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout 2023-11-14T07:44:53.5291998Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_nominal 2023-11-14T07:44:53.5293453Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client 2023-11-14T07:44:53.5294982Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set 2023-11-14T07:44:53.5296451Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv 2023-11-14T07:44:53.5297847Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_append_roundtrip 2023-11-14T07:44:53.5299258Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks 2023-11-14T07:44:53.5300752Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough 2023-11-14T07:44:53.5302217Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip 2023-11-14T07:44:53.5303639Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip 2023-11-14T07:44:53.5305141Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip 2023-11-14T07:44:53.5306558Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail 2023-11-14T07:44:53.5307924Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_0 2023-11-14T07:44:53.5309259Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_1 2023-11-14T07:44:53.5310592Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_2 2023-11-14T07:44:53.5311909Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_3 2023-11-14T07:44:53.5313235Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_4 2023-11-14T07:44:53.5314565Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_5 2023-11-14T07:44:53.5315893Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_6 2023-11-14T07:44:53.5317305Z Running 1 items in this shard: test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait 2023-11-14T07:44:53.5318692Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithUvStore::test_with_env_var 2023-11-14T07:44:53.5320028Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithUvStore::test_with_url_param 2023-11-14T07:44:53.5320819Z 2023-11-14T07:44:53.5321423Z Running distributed/fsdp/test_fsdp_hybrid_shard 1/1 ... [2023-11-14 07:44:53.522007] 2023-11-14T07:44:53.5323872Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:44:53.522263] 2023-11-14T07:45:44.1322971Z 2023-11-14T07:45:44.1324899Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_b53da69f833cc4c6_.log 2023-11-14T07:45:44.1334622Z Running 7 items in this shard: test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_basic_setup, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_parity, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_save_load_state_dict, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_sync_module_state, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hybrid_shard_pg_mismatch_raises, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_invalid_pg_specification_raises, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_raises_manual_wrap_hybrid_shard_when_none_policy 2023-11-14T07:45:44.1339783Z 2023-11-14T07:45:44.1340358Z Running distributed/fsdp/test_fsdp_comm 1/1 ... [2023-11-14 07:45:44.132486] 2023-11-14T07:45:44.1342749Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:45:44.132744] 2023-11-14T07:46:30.7371476Z 2023-11-14T07:46:30.7375805Z distributed/fsdp/test_fsdp_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_b0ad30105458d6dc_.log 2023-11-14T07:46:30.7385303Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP 2023-11-14T07:46:30.7393345Z 2023-11-14T07:46:30.7393900Z Running distributed/fsdp/test_fsdp_overlap 1/1 ... [2023-11-14 07:46:30.737323] 2023-11-14T07:46:30.7396325Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:46:30.737577] 2023-11-14T07:47:20.7419465Z 2023-11-14T07:47:20.7421603Z distributed/fsdp/test_fsdp_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_8f67f640c7fd6813_.log 2023-11-14T07:47:20.7424453Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeOne::test_forward_overlap, test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeTwo::test_forward_overlap 2023-11-14T07:47:20.7426127Z 2023-11-14T07:47:20.7426690Z Running distributed/checkpoint/test_checkpoint 1/1 ... [2023-11-14 07:47:20.742117] 2023-11-14T07:47:20.7429209Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:47:20.742407] 2023-11-14T07:48:01.4986350Z 2023-11-14T07:48:01.4988239Z distributed/checkpoint/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_checkpoint_1.1_23212d6d52d68fff_.log 2023-11-14T07:48:01.4996032Z Running 8 items in this shard: test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_default_metadata, test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_tensor_metadata_with_missing_rank_spec, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_reader_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_writer_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling_no_dist, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling_no_dist 2023-11-14T07:48:01.5001779Z 2023-11-14T07:48:01.5002322Z Running distributed/_tensor/test_matrix_ops 1/1 ... [2023-11-14 07:48:01.498927] 2023-11-14T07:48:01.5004714Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:48:01.499239] 2023-11-14T07:48:40.3412611Z 2023-11-14T07:48:40.3418498Z distributed/_tensor/test_matrix_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_matrix_ops_1.1_1e5fa2fe17809d44_.log 2023-11-14T07:48:40.3423447Z Running 7 items in this shard: test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_auto_redistribute, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_baddbmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_bmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_mm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial 2023-11-14T07:48:40.3427168Z 2023-11-14T07:48:40.3427658Z Running distributed/_tensor/test_init 1/1 ... [2023-11-14 07:48:40.341453] 2023-11-14T07:48:40.3429976Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_init.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:48:40.341735] 2023-11-14T07:49:11.0702702Z 2023-11-14T07:49:11.0704766Z distributed/_tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_init_1.1_7b115b83f4a0cec4_.log 2023-11-14T07:49:11.0712302Z Running 7 items in this shard: test/distributed/_tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh 2023-11-14T07:49:11.0716164Z 2023-11-14T07:49:11.0716783Z Running distributed/test_c10d_functional_native 1/1 ... [2023-11-14 07:49:11.070440] 2023-11-14T07:49:11.0719231Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:49:11.070720] 2023-11-14T07:49:39.3929750Z 2023-11-14T07:49:39.3932056Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_72717c95532b386d_.log 2023-11-14T07:49:39.3946673Z Running 15 items in this shard: test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_all_gather_into_tensor, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_all_reduce, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_all_reduce_, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_reduce_scatter_tensor, test/distributed/test_c10d_functional_native.py::C10DFunctionalNativeTest::test_reduce_scatter_tensor_coalesced 2023-11-14T07:49:39.3957667Z 2023-11-14T07:49:39.3958336Z Running distributed/checkpoint/test_dtensor_resharding 1/1 ... [2023-11-14 07:49:39.393254] 2023-11-14T07:49:39.3960925Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_resharding.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:49:39.393598] 2023-11-14T07:50:08.5136817Z 2023-11-14T07:50:08.5140846Z distributed/checkpoint/test_dtensor_resharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_3b4526746de6eac5_.log 2023-11-14T07:50:08.5145823Z Running 4 items in this shard: test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change 2023-11-14T07:50:08.5149480Z 2023-11-14T07:50:08.5150096Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2023-11-14 07:50:08.513819] 2023-11-14T07:50:08.5152730Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:50:08.514079] 2023-11-14T07:50:35.7802789Z 2023-11-14T07:50:35.7804758Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_c171de1fae27011c_.log 2023-11-14T07:50:35.7814307Z Running 5 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_True, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_True 2023-11-14T07:50:35.7819377Z 2023-11-14T07:50:35.7820001Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2023-11-14 07:50:35.780421] 2023-11-14T07:50:35.7822518Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:50:35.780680] 2023-11-14T07:50:58.1378642Z 2023-11-14T07:50:58.1380733Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_cb765f93f2a202cb_.log 2023-11-14T07:50:58.1386810Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False), test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True), test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False), test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) 2023-11-14T07:50:58.1391384Z 2023-11-14T07:50:58.1391907Z Running distributed/test_c10d_spawn_ucc 1/1 ... [2023-11-14 07:50:58.138034] 2023-11-14T07:50:58.1394349Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '--shard-id=0', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:50:58.138296] 2023-11-14T07:51:38.3205521Z 2023-11-14T07:51:38.3211053Z distributed/test_c10d_spawn_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_ucc_1.1_f38cf3c1188558dc_.log 2023-11-14T07:51:38.3213192Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather 2023-11-14T07:51:38.3215209Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all 2023-11-14T07:51:38.3216835Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single 2023-11-14T07:51:38.3218471Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce 2023-11-14T07:51:38.3220056Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast 2023-11-14T07:51:38.3221644Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce 2023-11-14T07:51:38.3222519Z 2023-11-14T07:51:38.3223045Z Running distributed/fsdp/test_fsdp_apply 1/1 ... [2023-11-14 07:51:38.320794] 2023-11-14T07:51:38.3225790Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:51:38.321062] 2023-11-14T07:51:58.8742072Z 2023-11-14T07:51:58.8743863Z distributed/fsdp/test_fsdp_apply 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_b66088cfc7293888_.log 2023-11-14T07:51:58.8747673Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_apply_in_summon_raises_error, test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_nested_module_apply, test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_transformer_module_apply 2023-11-14T07:51:58.8749762Z 2023-11-14T07:51:58.8750424Z Running distributed/checkpoint/test_state_dict_utils 1/1 ... [2023-11-14 07:51:58.874441] 2023-11-14T07:51:58.8752973Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_state_dict_utils.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:51:58.874702] 2023-11-14T07:52:15.1217675Z 2023-11-14T07:52:15.1219878Z distributed/checkpoint/test_state_dict_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_5ab508dd65919998_.log 2023-11-14T07:52:15.1239042Z Running 4 items in this shard: test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_complicated_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_cpu_and_ranks_only, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_state_dict_dtensor, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_with_cpu_and_ranks_only 2023-11-14T07:52:15.1242205Z 2023-11-14T07:52:15.1242894Z Running distributed/checkpoint/test_fsdp_optim_state 1/1 ... [2023-11-14 07:52:15.121908] 2023-11-14T07:52:15.1245449Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:52:15.122183] 2023-11-14T07:52:30.3161533Z 2023-11-14T07:52:30.3163737Z distributed/checkpoint/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_3a8de01c3537b5ac_.log 2023-11-14T07:52:30.3167780Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_False, test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_True 2023-11-14T07:52:30.3169919Z 2023-11-14T07:52:30.3170551Z Running distributed/checkpoint/test_fsdp_model_state 1/1 ... [2023-11-14 07:52:30.316378] 2023-11-14T07:52:30.3173486Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:52:30.316658] 2023-11-14T07:52:44.4578523Z 2023-11-14T07:52:44.4580766Z distributed/checkpoint/test_fsdp_model_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_0a94a8c2fd39d352_.log 2023-11-14T07:52:44.4584002Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_no_resharding, test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_with_resharding 2023-11-14T07:52:44.4586298Z 2023-11-14T07:52:44.4586926Z Running distributed/_shard/sharded_tensor/ops/test_init 1/1 ... [2023-11-14 07:52:44.457981] 2023-11-14T07:52:44.4589495Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:52:44.458254] 2023-11-14T07:52:56.0474352Z 2023-11-14T07:52:56.0476562Z distributed/_shard/sharded_tensor/ops/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_f769f941e7e30c94_.log 2023-11-14T07:52:56.0480565Z Running 3 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_kaiming_uniform, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_normal, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_uniform 2023-11-14T07:52:56.0483259Z 2023-11-14T07:52:56.0483885Z Running distributed/elastic/timer/local_timer_example 1/1 ... [2023-11-14 07:52:56.047673] 2023-11-14T07:52:56.0486476Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:52:56.047941] 2023-11-14T07:53:06.6371578Z 2023-11-14T07:53:06.6376027Z distributed/elastic/timer/local_timer_example 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_example_1.1_1588dbbc2ee073d6_.log 2023-11-14T07:53:06.6379085Z Running 2 items in this shard: test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_example_start_method_spawn, test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_torch_mp_example 2023-11-14T07:53:06.6380810Z 2023-11-14T07:53:06.6381519Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 ... [2023-11-14 07:53:06.637358] 2023-11-14T07:53:06.6384244Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:53:06.637630] 2023-11-14T07:53:15.3752253Z 2023-11-14T07:53:15.3754978Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_reshard_1.1_f0f159d2b6036333_.log 2023-11-14T07:53:15.3758539Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard, test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard_errors 2023-11-14T07:53:15.3760313Z 2023-11-14T07:53:15.3761030Z Running distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2023-11-14 07:53:15.375360] 2023-11-14T07:53:15.3763745Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:53:15.375628] 2023-11-14T07:53:25.2119522Z 2023-11-14T07:53:25.2122283Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_c79d50a680af6e2a_.log 2023-11-14T07:53:25.2125299Z Running 1 items in this shard: test/distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py::TestFsdpTpCheckpointConversion::test_fsdp_to_tp 2023-11-14T07:53:25.2126407Z 2023-11-14T07:53:25.2127061Z Running distributed/checkpoint/test_dtensor_checkpoint 1/1 ... [2023-11-14 07:53:25.212161] 2023-11-14T07:53:25.2129619Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_checkpoint.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:53:25.212475] 2023-11-14T07:53:34.9458730Z 2023-11-14T07:53:34.9461047Z distributed/checkpoint/test_dtensor_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_476c72372bdc71cd_.log 2023-11-14T07:53:34.9463427Z Running 1 items in this shard: test/distributed/checkpoint/test_dtensor_checkpoint.py::DTensorPlanner::test_distributed_tensor_planner 2023-11-14T07:53:34.9464489Z 2023-11-14T07:53:34.9469301Z Running distributed/fsdp/test_fsdp_traversal 1/1 ... [2023-11-14 07:53:34.946575] 2023-11-14T07:53:34.9474108Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:53:34.946948] 2023-11-14T07:53:44.5800015Z 2023-11-14T07:53:44.5801849Z distributed/fsdp/test_fsdp_traversal 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_8057e0e65c3589cb_.log 2023-11-14T07:53:44.5804268Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_traversal.py::TestTraversal::test_fsdp_modules 2023-11-14T07:53:44.5805485Z 2023-11-14T07:53:44.5807678Z Running distributed/fsdp/test_fsdp_multiple_wrapping 1/1 ... [2023-11-14 07:53:44.580333] 2023-11-14T07:53:44.5811856Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:53:44.580678] 2023-11-14T07:53:53.8633493Z 2023-11-14T07:53:53.8635853Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_dbf4284801bb1e2b_.log 2023-11-14T07:53:53.8638616Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_wrapping.py::TestMultipleWrapping::test_multiple_wrapping 2023-11-14T07:53:53.8639644Z 2023-11-14T07:53:53.8640730Z Running distributed/elastic/timer/local_timer_test 1/1 ... [2023-11-14 07:53:53.863657] 2023-11-14T07:53:53.8644517Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:53:53.863981] 2023-11-14T07:54:00.2414939Z 2023-11-14T07:54:00.2417105Z distributed/elastic/timer/local_timer_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_test_1.1_22941d1297b02b8a_.log 2023-11-14T07:54:00.2428560Z Running 14 items in this shard: test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_client_interaction, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_exception_propagation, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_get_timer_recursive, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_happy_path, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_no_client, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_timer, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_less_than_size, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_size, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_acquire_release, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_expired_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_valid_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_call_count, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_empty_queue 2023-11-14T07:54:00.2438393Z 2023-11-14T07:54:00.2438954Z Running distributed/rpc/test_share_memory 1/1 ... [2023-11-14 07:54:00.241800] 2023-11-14T07:54:00.2441467Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '--shard-id=0', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:00.242121] 2023-11-14T07:54:07.6333241Z 2023-11-14T07:54:07.6334994Z distributed/rpc/test_share_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_share_memory_1.1_1b953cb39c46370b_.log 2023-11-14T07:54:07.6337172Z Running 1 items in this shard: test/distributed/rpc/test_share_memory.py::TestRPCPickler::test_case 2023-11-14T07:54:07.6338245Z 2023-11-14T07:54:07.6339294Z Running distributed/pipeline/sync/test_stream 1/1 ... [2023-11-14 07:54:07.633550] 2023-11-14T07:54:07.6343550Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_stream.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:07.633870] 2023-11-14T07:54:11.0548423Z 2023-11-14T07:54:11.0550257Z distributed/pipeline/sync/test_stream 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_stream_1.1_64525298bc454ca2_.log 2023-11-14T07:54:11.0563554Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view 2023-11-14T07:54:11.0575964Z 2023-11-14T07:54:11.0576567Z Running distributed/pipeline/sync/test_copy 1/1 ... [2023-11-14 07:54:11.055216] 2023-11-14T07:54:11.0579067Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_copy.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:11.055535] 2023-11-14T07:54:14.3264319Z 2023-11-14T07:54:14.3266429Z distributed/pipeline/sync/test_copy 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_copy_1.1_314456196a3318fc_.log 2023-11-14T07:54:14.3270344Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors 2023-11-14T07:54:14.3273084Z 2023-11-14T07:54:14.3273670Z Running distributed/pipeline/sync/skip/test_leak 1/1 ... [2023-11-14 07:54:14.326718] 2023-11-14T07:54:14.3276207Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_leak.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:14.327030] 2023-11-14T07:54:17.1972176Z 2023-11-14T07:54:17.1974743Z distributed/pipeline/sync/skip/test_leak 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_leak_1.1_9d12d5f6fecf9550_.log 2023-11-14T07:54:17.1982410Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] 2023-11-14T07:54:17.1987617Z 2023-11-14T07:54:17.1988150Z Running distributed/_tensor/test_op_strategy 1/1 ... [2023-11-14 07:54:17.197444] 2023-11-14T07:54:17.1990562Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_op_strategy.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:17.197720] 2023-11-14T07:54:21.4202888Z 2023-11-14T07:54:21.4204908Z distributed/_tensor/test_op_strategy 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_op_strategy_1.1_0ea0f8fd21c07057_.log 2023-11-14T07:54:21.4214198Z Running 14 items in this shard: test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_batch_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_bmm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_free_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_mm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_linearity_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_pointwise_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_bmm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_mm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_1d, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_2d 2023-11-14T07:54:21.4222501Z 2023-11-14T07:54:21.4223096Z Running distributed/elastic/events/lib_test 1/1 ... [2023-11-14 07:54:21.420504] 2023-11-14T07:54:21.4225563Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/events/lib_test.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:21.420802] 2023-11-14T07:54:23.9403281Z 2023-11-14T07:54:23.9405168Z distributed/elastic/events/lib_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.events.lib_test_1.1_cb7840f0821b4016_.log 2023-11-14T07:54:23.9413062Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2023-11-14T07:54:23.9418240Z 2023-11-14T07:54:23.9418830Z Running distributed/nn/jit/test_instantiator 1/1 ... [2023-11-14 07:54:23.940522] 2023-11-14T07:54:23.9421279Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:23.940820] 2023-11-14T07:54:26.3600531Z 2023-11-14T07:54:26.3602586Z distributed/nn/jit/test_instantiator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.nn.jit.test_instantiator_1.1_f9a32daeb029829c_.log 2023-11-14T07:54:26.3606900Z Running 3 items in this shard: test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_get_arg_return_types_from_interface, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_non_scripted_remote_module_template, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_scripted_remote_module_template 2023-11-14T07:54:26.3609964Z 2023-11-14T07:54:26.3610601Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2023-11-14 07:54:26.360301] 2023-11-14T07:54:26.3613651Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:26.360595] 2023-11-14T07:54:28.7798238Z 2023-11-14T07:54:28.7800165Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_2f3711ec3f23f16b_.log 2023-11-14T07:54:28.7803264Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2023-11-14T07:54:28.7804300Z 2023-11-14T07:54:28.7805134Z Running distributed/checkpoint/test_traverse 1/1 ... [2023-11-14 07:54:28.779999] 2023-11-14T07:54:28.7807977Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:28.780289] 2023-11-14T07:54:33.4534978Z 2023-11-14T07:54:33.4537021Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_a8c3aa361caa42e3_.log 2023-11-14T07:54:33.4543307Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict 2023-11-14T07:54:33.4547903Z 2023-11-14T07:54:33.4548513Z Running distributed/_composable/test_checkpoint 1/1 ... [2023-11-14 07:54:33.453720] 2023-11-14T07:54:33.4550971Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:33.454032] 2023-11-14T07:54:38.5780534Z 2023-11-14T07:54:38.5782941Z distributed/_composable/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_checkpoint_1.1_029f98d9627b9fb2_.log 2023-11-14T07:54:38.5788229Z Running 5 items in this shard: test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_clears_state_on_error_in_forward, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_multi_args, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_random_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_gpu 2023-11-14T07:54:38.5791305Z 2023-11-14T07:54:38.5791952Z Running distributed/pipeline/sync/skip/test_portal 1/1 ... [2023-11-14 07:54:38.578293] 2023-11-14T07:54:38.5794503Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_portal.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:38.578584] 2023-11-14T07:54:40.7973885Z 2023-11-14T07:54:40.7976186Z distributed/pipeline/sync/skip/test_portal 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_portal_1.1_00960e18abe2ded1_.log 2023-11-14T07:54:40.7984003Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 2023-11-14T07:54:40.7990401Z 2023-11-14T07:54:40.7990974Z Running distributed/pipeline/sync/test_phony 1/1 ... [2023-11-14 07:54:40.797659] 2023-11-14T07:54:40.7993439Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_phony.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:40.797956] 2023-11-14T07:54:42.9167227Z 2023-11-14T07:54:42.9169279Z distributed/pipeline/sync/test_phony 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_phony_1.1_e37b13b21d9d4804_.log 2023-11-14T07:54:42.9173180Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function 2023-11-14T07:54:42.9175422Z 2023-11-14T07:54:42.9175957Z Running distributed/_spmd/test_graph_utils 1/1 ... [2023-11-14 07:54:42.916927] 2023-11-14T07:54:42.9178345Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_spmd/test_graph_utils.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:42.917274] 2023-11-14T07:54:46.6306742Z 2023-11-14T07:54:46.6308512Z distributed/_spmd/test_graph_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_graph_utils_1.1_448000892674d5c9_.log 2023-11-14T07:54:46.6310089Z 2023-11-14T07:54:46.6311721Z Running distributed/_tensor/test_optimizers 1/1 ... [2023-11-14 07:54:46.630838] 2023-11-14T07:54:46.6316005Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_optimizers.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:46.631129] 2023-11-14T07:54:50.2296865Z 2023-11-14T07:54:50.2298626Z distributed/_tensor/test_optimizers 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_optimizers_1.1_946b5e06fb690ac7_.log 2023-11-14T07:54:50.2300325Z 2023-11-14T07:54:50.2302740Z Running distributed/optim/test_apply_optimizer_in_backward 1/1 ... [2023-11-14 07:54:50.229884] 2023-11-14T07:54:50.2306291Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:50.230180] 2023-11-14T07:54:54.6452017Z 2023-11-14T07:54:54.6453963Z distributed/optim/test_apply_optimizer_in_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_751d3fc6290fe2ea_.log 2023-11-14T07:54:54.6456027Z 2023-11-14T07:54:54.6457240Z Running distributed/test_collective_utils 1/1 ... [2023-11-14 07:54:54.645392] 2023-11-14T07:54:54.6461488Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:54:54.645692] 2023-11-14T07:54:57.1692880Z 2023-11-14T07:54:57.1694849Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_2bb22d83665ed083_.log 2023-11-14T07:54:57.1696177Z 2023-11-14T07:54:57.1698650Z Starting test batch 'unlikely_relevance' 10776.695002555847 seconds after initiating testing 2023-11-14T07:54:57.1699918Z With sharding, this batch will run 0 tests 2023-11-14T07:54:57.1700885Z Starting test batch 'none_relevance' 10776.695046424866 seconds after initiating testing 2023-11-14T07:54:57.1701769Z With sharding, this batch will run 0 tests 2023-11-14T07:54:58.2383862Z /var/lib/jenkins/workspace/tools/stats/upload_metrics.py:164: UserWarning: Error uploading metric td_experiment_1 to DynamoDB: Unsupported type "" for value "distributed/test_distributed_spawn" 2023-11-14T07:54:58.2386059Z warn(f"Error uploading metric {metric_name} to DynamoDB: {e}") 2023-11-14T07:54:58.2990293Z Emiting td_test_failure_stats 2023-11-14T07:54:59.7217257Z distributed/test_distributed_spawn 1/2 failed! 2023-11-14T07:54:59.9872497Z 2023-11-14T07:54:59.9873202Z real 179m41.927s 2023-11-14T07:54:59.9873665Z user 245m21.524s 2023-11-14T07:54:59.9874040Z sys 30m22.894s 2023-11-14T07:54:59.9923254Z ##[error]Process completed with exit code 1. 2023-11-14T07:55:00.0018662Z Prepare all required actions 2023-11-14T07:55:00.0019254Z Getting action download info 2023-11-14T07:55:00.1608977Z ##[group]Run ./.github/actions/pytest-cache-upload 2023-11-14T07:55:00.1609574Z with: 2023-11-14T07:55:00.1609912Z cache_dir: .pytest_cache 2023-11-14T07:55:00.1610365Z shard: 1 2023-11-14T07:55:00.1611622Z job_identifier: pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed 2023-11-14T07:55:00.1612932Z env: 2023-11-14T07:55:00.1613391Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:00.1613986Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:00.1614978Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:00.1615819Z ##[endgroup] 2023-11-14T07:55:00.1668852Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T07:55:00.1669554Z with: 2023-11-14T07:55:00.1669868Z shell: bash 2023-11-14T07:55:00.1670230Z timeout_minutes: 5 2023-11-14T07:55:00.1670708Z max_attempts: 5 2023-11-14T07:55:00.1671091Z retry_wait_seconds: 30 2023-11-14T07:55:00.1671658Z command: set -eu python3 -m pip install boto3==1.19.12 2023-11-14T07:55:00.1672308Z polling_interval_seconds: 1 2023-11-14T07:55:00.1672749Z warning_on_retry: true 2023-11-14T07:55:00.1673180Z continue_on_error: false 2023-11-14T07:55:00.1673593Z env: 2023-11-14T07:55:00.1673905Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:00.1674463Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:00.1675386Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:00.1676173Z ##[endgroup] 2023-11-14T07:55:00.5344339Z Defaulting to user installation because normal site-packages is not writeable 2023-11-14T07:55:00.5561292Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12) 2023-11-14T07:55:00.5633752Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12) 2023-11-14T07:55:00.5698990Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0) 2023-11-14T07:55:00.5718647Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2) 2023-11-14T07:55:00.5756883Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18) 2023-11-14T07:55:00.6006556Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2) 2023-11-14T07:55:00.6035628Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2023-11-14T07:55:01.2387446Z Command completed after 1 attempt(s). 2023-11-14T07:55:01.2444064Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2023-11-14T07:55:01.2444804Z python3 .github/scripts/pytest_cache.py \ 2023-11-14T07:55:01.2445381Z  --upload \ 2023-11-14T07:55:01.2445894Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2023-11-14T07:55:01.2446531Z  --pr_identifier $GITHUB_REF \ 2023-11-14T07:55:01.2447105Z  --job_identifier $JOB_IDENTIFIER \ 2023-11-14T07:55:01.2447665Z  --shard $SHARD \ 2023-11-14T07:55:01.2448121Z  --repo $REPO \ 2023-11-14T07:55:01.2448563Z  --temp_dir $RUNNER_TEMP \ 2023-11-14T07:55:01.2457972Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:55:01.2458604Z env: 2023-11-14T07:55:01.2458943Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:01.2459490Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:01.2460593Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:01.2461467Z CACHE_DIR: .pytest_cache 2023-11-14T07:55:01.2462113Z JOB_IDENTIFIER: pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed 2023-11-14T07:55:01.2462827Z SHARD: 1 2023-11-14T07:55:01.2463180Z REPO: pytorch/pytorch 2023-11-14T07:55:01.2463576Z ##[endgroup] 2023-11-14T07:55:01.6936283Z PR identifier for `refs/pull/112098/merge` is `2e5132e7254b4b60102d80422aad461b` 2023-11-14T07:55:01.6939603Z Uploading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=False, job_identifier='pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed', pr_identifier='refs/pull/112098/merge', repo='pytorch/pytorch', shard='1', temp_dir='/home/ec2-user/actions-runner/_work/_temp', upload=True) 2023-11-14T07:55:01.6942499Z Zipping /home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache 2023-11-14T07:55:01.6944427Z to /home/ec2-user/actions-runner/_work/_temp/zip-upload/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1 2023-11-14T07:55:01.6947142Z Uploading /home/ec2-user/actions-runner/_work/_temp/zip-upload/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1.zip 2023-11-14T07:55:01.6949614Z to s3://gha-artifacts/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1.zip 2023-11-14T07:55:01.7317619Z ##[group]Run cat test/**/*_toprint.log || true 2023-11-14T07:55:01.7318292Z cat test/**/*_toprint.log || true 2023-11-14T07:55:01.7326901Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:55:01.7327524Z env: 2023-11-14T07:55:01.7327845Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:01.7328409Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:01.7329339Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:01.7330148Z ##[endgroup] 2023-11-14T07:55:01.7376643Z cat: test/**/*_toprint.log: No such file or directory 2023-11-14T07:55:01.7415204Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2023-11-14T07:55:01.7415830Z kill "$MONITOR_SCRIPT_PID" 2023-11-14T07:55:01.7423848Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:55:01.7424475Z env: 2023-11-14T07:55:01.7424793Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:01.7425353Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:01.7426285Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:01.7427098Z MONITOR_SCRIPT_PID: 85411 2023-11-14T07:55:01.7427535Z ##[endgroup] 2023-11-14T07:55:01.7637673Z Prepare all required actions 2023-11-14T07:55:01.7638277Z Getting action download info 2023-11-14T07:55:01.9033170Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2023-11-14T07:55:02.0994169Z ##[group]Run ./.github/actions/upload-test-artifacts 2023-11-14T07:55:02.0994974Z with: 2023-11-14T07:55:02.0995580Z file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019 2023-11-14T07:55:02.0996319Z env: 2023-11-14T07:55:02.0996633Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:02.0997197Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:02.0998117Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:02.0998911Z ##[endgroup] 2023-11-14T07:55:02.1039694Z ##[group]Run # Remove any previous test jsons if they exist 2023-11-14T07:55:02.1040474Z # Remove any previous test jsons if they exist 2023-11-14T07:55:02.1041176Z rm -f test-jsons-*.zip 2023-11-14T07:55:02.1041818Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2023-11-14T07:55:02.1050469Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:55:02.1051360Z env: 2023-11-14T07:55:02.1051709Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:02.1052270Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:02.1053211Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:02.1054288Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019 2023-11-14T07:55:02.1055016Z ##[endgroup] 2023-11-14T07:55:02.1199614Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2023-11-14T07:55:02.1234863Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2023-11-14T07:55:02.1235756Z adding: test/minioptest_failures_dict.json (deflated 70%) 2023-11-14T07:55:02.1242892Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2023-11-14T07:55:02.1246093Z adding: test/.pytorch-slow-tests.json (deflated 78%) 2023-11-14T07:55:02.1254311Z adding: test/.pytorch-disabled-tests.json (deflated 85%) 2023-11-14T07:55:02.1288010Z ##[group]Run # Remove any previous test reports if they exist 2023-11-14T07:55:02.1288822Z # Remove any previous test reports if they exist 2023-11-14T07:55:02.1289496Z rm -f test-reports-*.zip 2023-11-14T07:55:02.1290234Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2023-11-14T07:55:02.1299137Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:55:02.1299740Z env: 2023-11-14T07:55:02.1300083Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:02.1300646Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:02.1301556Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:02.1302634Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019 2023-11-14T07:55:02.1303396Z ##[endgroup] 2023-11-14T07:55:02.1483577Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8876298fb8fe2bfe.xml (deflated 39%) 2023-11-14T07:55:02.1485669Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e858d2107f0cf12.xml (deflated 44%) 2023-11-14T07:55:02.1487750Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65c5c9ec36064c07.xml (deflated 38%) 2023-11-14T07:55:02.1489811Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f4646f64f4bfd691.xml (deflated 38%) 2023-11-14T07:55:02.1492237Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d50f65c16a956690.xml (deflated 38%) 2023-11-14T07:55:02.1494492Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-716ecc4844ea9253.xml (deflated 38%) 2023-11-14T07:55:02.1496607Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6782437599197a1.xml (deflated 37%) 2023-11-14T07:55:02.1498659Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41a68ce5c5173766.xml (deflated 37%) 2023-11-14T07:55:02.1500899Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2104300859c4f6b7.xml (deflated 37%) 2023-11-14T07:55:02.1502927Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e442951353ad2b36.xml (deflated 43%) 2023-11-14T07:55:02.1504973Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4cdf1030c314a7a5.xml (deflated 46%) 2023-11-14T07:55:02.1507032Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf5f528072c57f75.xml (deflated 46%) 2023-11-14T07:55:02.1509083Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ffd6a01fe4576ccf.xml (deflated 45%) 2023-11-14T07:55:02.1511114Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24624f325874c6fe.xml (deflated 38%) 2023-11-14T07:55:02.1513175Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae1a2f724418face.xml (deflated 37%) 2023-11-14T07:55:02.1515223Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0702e86fa5f3cd07.xml (deflated 37%) 2023-11-14T07:55:02.1517266Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2264f07f07e90be9.xml (deflated 37%) 2023-11-14T07:55:02.1519302Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ddd68af45b0966d.xml (deflated 37%) 2023-11-14T07:55:02.1521356Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1ebe63c1afb4f28.xml (deflated 44%) 2023-11-14T07:55:02.1523416Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5157ec5defc524b.xml (deflated 44%) 2023-11-14T07:55:02.1525472Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f06805f53d07374e.xml (deflated 44%) 2023-11-14T07:55:02.1527499Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fb39f25b77c5a373.xml (deflated 44%) 2023-11-14T07:55:02.1529551Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc6fa05d91772b0f.xml (deflated 44%) 2023-11-14T07:55:02.1531889Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e9f7a87feb1ef5c.xml (deflated 44%) 2023-11-14T07:55:02.1533946Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24efa0936b18333d.xml (deflated 43%) 2023-11-14T07:55:02.1536042Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98cff233cd45cc86.xml (deflated 43%) 2023-11-14T07:55:02.1538076Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a37541e9cbf76b3.xml (deflated 43%) 2023-11-14T07:55:02.1540126Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38c01fcac6787ac4.xml (deflated 43%) 2023-11-14T07:55:02.1542186Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ee1db3e9469dc51.xml (deflated 42%) 2023-11-14T07:55:02.1544373Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-879111c405d6a258.xml (deflated 43%) 2023-11-14T07:55:02.1546429Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00e2b330c490f065.xml (deflated 43%) 2023-11-14T07:55:02.1548624Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-672f6a210282fad9.xml (deflated 43%) 2023-11-14T07:55:02.1550673Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b24aa410c29d051.xml (deflated 43%) 2023-11-14T07:55:02.1552716Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d645e7dfba55256f.xml (deflated 43%) 2023-11-14T07:55:02.1554747Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44bea51ed5a714a8.xml (deflated 38%) 2023-11-14T07:55:02.1556790Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1283617f483b5581.xml (deflated 37%) 2023-11-14T07:55:02.1558828Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f4432c0b07831f7.xml (deflated 43%) 2023-11-14T07:55:02.1560890Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d7f54792907db84.xml (deflated 38%) 2023-11-14T07:55:02.1562904Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef8d89339d73f81e.xml (deflated 38%) 2023-11-14T07:55:02.1564947Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e1d5c3485c9857f.xml (deflated 38%) 2023-11-14T07:55:02.1566994Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0caaf776ae254cd7.xml (deflated 49%) 2023-11-14T07:55:02.1569051Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2f0c82d0e85077b.xml (deflated 44%) 2023-11-14T07:55:02.1571602Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c77df20357375f7.xml (deflated 44%) 2023-11-14T07:55:02.1573707Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4202c56ce6fb5744.xml (deflated 37%) 2023-11-14T07:55:02.1575755Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e0d67e82e894e7a2.xml (deflated 38%) 2023-11-14T07:55:02.1577803Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30aca758b6c52f7f.xml (deflated 37%) 2023-11-14T07:55:02.1579821Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7a61f8fc65935b5.xml (deflated 44%) 2023-11-14T07:55:02.1581871Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee6e6c457e0601b6.xml (deflated 48%) 2023-11-14T07:55:02.1583978Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d56ed12eb47f185b.xml (deflated 37%) 2023-11-14T07:55:02.1586035Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f504d35cfbb6b4c.xml (deflated 37%) 2023-11-14T07:55:02.1588077Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c0957bdf3b5b173.xml (deflated 44%) 2023-11-14T07:55:02.1590120Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-725b262fc058b06d.xml (deflated 38%) 2023-11-14T07:55:02.1592167Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e229253229760cb.xml (deflated 44%) 2023-11-14T07:55:02.1594354Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-085939aaf3006c8b.xml (deflated 37%) 2023-11-14T07:55:02.1596422Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e35af27529fa64d2.xml (deflated 43%) 2023-11-14T07:55:02.1598610Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a5bc7c7dfb8a3f1.xml (deflated 44%) 2023-11-14T07:55:02.1600664Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0aec5dce42559a3.xml (deflated 44%) 2023-11-14T07:55:02.1602717Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eaa44bf015490ae2.xml (deflated 38%) 2023-11-14T07:55:02.1604741Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-182a7f36c14b90db.xml (deflated 37%) 2023-11-14T07:55:02.1606799Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a54e3a5b6373801f.xml (deflated 37%) 2023-11-14T07:55:02.1608852Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d8c2b69d26446182.xml (deflated 43%) 2023-11-14T07:55:02.1611145Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45eca4f836bae382.xml (deflated 43%) 2023-11-14T07:55:02.1613198Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96d787fa81f163f9.xml (deflated 43%) 2023-11-14T07:55:02.1615252Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8054abf46ee6005.xml (deflated 45%) 2023-11-14T07:55:02.1617307Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89ab2dedf5b0b015.xml (deflated 38%) 2023-11-14T07:55:02.1619361Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-106ffe4f54c7fa71.xml (deflated 37%) 2023-11-14T07:55:02.1621397Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e7dfcd7a2e95c261.xml (deflated 37%) 2023-11-14T07:55:02.1623449Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-881184d5f2abdf40.xml (deflated 38%) 2023-11-14T07:55:02.1625500Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7faf74063ec9b28e.xml (deflated 37%) 2023-11-14T07:55:02.1627540Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f524579f4107c213.xml (deflated 57%) 2023-11-14T07:55:02.1629571Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-856339ea6a579b82.xml (deflated 38%) 2023-11-14T07:55:02.1631606Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fcc0e938007aab25.xml (deflated 38%) 2023-11-14T07:55:02.1633651Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39059549952c84cf.xml (deflated 37%) 2023-11-14T07:55:02.1635701Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b04afada30d770c.xml (deflated 37%) 2023-11-14T07:55:02.1637735Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8238bc2bcc551ed1.xml (deflated 38%) 2023-11-14T07:55:02.1639783Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cffbe971aafc21a.xml (deflated 37%) 2023-11-14T07:55:02.1641956Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70a23b7a9c9eff04.xml (deflated 37%) 2023-11-14T07:55:02.1644035Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d69c11423011e92d.xml (deflated 37%) 2023-11-14T07:55:02.1646061Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-871daa7ba0146ddb.xml (deflated 38%) 2023-11-14T07:55:02.1648232Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1d4e79264750edf.xml (deflated 37%) 2023-11-14T07:55:02.1650294Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f60a9bce2947d5e.xml (deflated 43%) 2023-11-14T07:55:02.1652522Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d046f0f065d8211.xml (deflated 37%) 2023-11-14T07:55:02.1654564Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05c41683ef296a76.xml (deflated 37%) 2023-11-14T07:55:02.1656605Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf48c6b090ece056.xml (deflated 38%) 2023-11-14T07:55:02.1658658Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0be3aa4010009984.xml (deflated 37%) 2023-11-14T07:55:02.1660700Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9614168087e4f056.xml (deflated 38%) 2023-11-14T07:55:02.1662727Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1e1adc474d0dbeb.xml (deflated 37%) 2023-11-14T07:55:02.1664780Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7e45b0b898c9abe.xml (deflated 37%) 2023-11-14T07:55:02.1666840Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e53cdf72bf119015.xml (deflated 38%) 2023-11-14T07:55:02.1668899Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-462bdc3ea1006f90.xml (deflated 37%) 2023-11-14T07:55:02.1670927Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6888312ac76af6a.xml (deflated 37%) 2023-11-14T07:55:02.1672985Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92bf115afc12bc42.xml (deflated 37%) 2023-11-14T07:55:02.1675031Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19856e7b55273f53.xml (deflated 49%) 2023-11-14T07:55:02.1677067Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7968b19129a02bc.xml (deflated 37%) 2023-11-14T07:55:02.1679109Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36c06cd37f28fc56.xml (deflated 38%) 2023-11-14T07:55:02.1681147Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3f4c4d163885b3e.xml (deflated 38%) 2023-11-14T07:55:02.1683242Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-06889df09ea8795f.xml (deflated 37%) 2023-11-14T07:55:02.1685289Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25c6a92195d875e3.xml (deflated 37%) 2023-11-14T07:55:02.1687313Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-834059860b17dae7.xml (deflated 37%) 2023-11-14T07:55:02.1689356Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d28fd7b6a9a4f70e.xml (deflated 43%) 2023-11-14T07:55:02.1692047Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e7638a3c561fcfbf.xml (deflated 43%) 2023-11-14T07:55:02.1694390Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8fc852cad02726ea.xml (deflated 37%) 2023-11-14T07:55:02.1696574Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-920352a7f971d088.xml (deflated 38%) 2023-11-14T07:55:02.1698623Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24ff2861aaed06f5.xml (deflated 37%) 2023-11-14T07:55:02.1700670Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38f9607d365f43b7.xml (deflated 43%) 2023-11-14T07:55:02.1702704Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-732407661991e376.xml (deflated 45%) 2023-11-14T07:55:02.1704720Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32061c572117305c.xml (deflated 44%) 2023-11-14T07:55:02.1706763Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e773f90cb9a119d6.xml (deflated 44%) 2023-11-14T07:55:02.1708822Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aedc4ce4ab9f4202.xml (deflated 38%) 2023-11-14T07:55:02.1710882Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ca692a6ad8e0e9c.xml (deflated 38%) 2023-11-14T07:55:02.1712919Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8697318c5e03a96e.xml (deflated 43%) 2023-11-14T07:55:02.1714964Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bfaac3914b749368.xml (deflated 43%) 2023-11-14T07:55:02.1716999Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-456e88b42ced9096.xml (deflated 37%) 2023-11-14T07:55:02.1719034Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e88225b8666e0673.xml (deflated 38%) 2023-11-14T07:55:02.1721081Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-74142999e0559237.xml (deflated 38%) 2023-11-14T07:55:02.1723106Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-94fee73785430646.xml (deflated 37%) 2023-11-14T07:55:02.1725128Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f1b330ef6364ae9.xml (deflated 36%) 2023-11-14T07:55:02.1727174Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f8d34e35f9b3398.xml (deflated 43%) 2023-11-14T07:55:02.1729214Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3749e0825a57a839.xml (deflated 43%) 2023-11-14T07:55:02.1731523Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c80f9dc6eba660a.xml (deflated 43%) 2023-11-14T07:55:02.1733578Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d42d5bfe23012457.xml (deflated 43%) 2023-11-14T07:55:02.1735626Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d75a865d5828abc.xml (deflated 38%) 2023-11-14T07:55:02.1737665Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9376e5b65d8f3a32.xml (deflated 38%) 2023-11-14T07:55:02.1739710Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d1cc47f38b75a5f.xml (deflated 43%) 2023-11-14T07:55:02.1741898Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c21bdcdda20f524a.xml (deflated 43%) 2023-11-14T07:55:02.1743987Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f19e5bb300fffa6e.xml (deflated 44%) 2023-11-14T07:55:02.1746172Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5917e678e958470.xml (deflated 46%) 2023-11-14T07:55:02.1748215Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d43523eceb6cc66.xml (deflated 45%) 2023-11-14T07:55:02.1750234Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b311975a47f53931.xml (deflated 45%) 2023-11-14T07:55:02.1752268Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f8818af4033e4627.xml (deflated 38%) 2023-11-14T07:55:02.1754323Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76a5904a4ee2068e.xml (deflated 46%) 2023-11-14T07:55:02.1756370Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a51789c80150bcbf.xml (deflated 45%) 2023-11-14T07:55:02.1758407Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f5049d52a8d5bd74.xml (deflated 43%) 2023-11-14T07:55:02.1760458Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d292cfedc3e0dee6.xml (deflated 37%) 2023-11-14T07:55:02.1762515Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7cfd4e0a430d23b.xml (deflated 38%) 2023-11-14T07:55:02.1764557Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1327049e3c9cb326.xml (deflated 38%) 2023-11-14T07:55:02.1766586Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9efedc85a37e721d.xml (deflated 38%) 2023-11-14T07:55:02.1768642Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f5f13ee46311949.xml (deflated 48%) 2023-11-14T07:55:02.1770857Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c35b0f1bbb33b2ab.xml (deflated 39%) 2023-11-14T07:55:02.1772939Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a55c3b896770f4b9.xml (deflated 44%) 2023-11-14T07:55:02.1774956Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3400a6578f6916ad.xml (deflated 38%) 2023-11-14T07:55:02.1776998Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36172e21fafec4bc.xml (deflated 38%) 2023-11-14T07:55:02.1779045Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-149ef7855262048e.xml (deflated 38%) 2023-11-14T07:55:02.1781073Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c836ed16a7834f05.xml (deflated 38%) 2023-11-14T07:55:02.1783198Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-333ede93ca78ced4.xml (deflated 37%) 2023-11-14T07:55:02.1785246Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fdc0e6f77d0162a4.xml (deflated 37%) 2023-11-14T07:55:02.1787286Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45720b6a2d4eb49e.xml (deflated 37%) 2023-11-14T07:55:02.1789454Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d35f880db51b3c82.xml (deflated 43%) 2023-11-14T07:55:02.1791508Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9b3ef47a8b3b864.xml (deflated 46%) 2023-11-14T07:55:02.1793557Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8381e2fe90cb7d2e.xml (deflated 46%) 2023-11-14T07:55:02.1795751Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ad2e84d01de3649.xml (deflated 45%) 2023-11-14T07:55:02.1797798Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-feee06041bba879a.xml (deflated 37%) 2023-11-14T07:55:02.1799820Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-383fb6b099e0d148.xml (deflated 37%) 2023-11-14T07:55:02.1801869Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5487748b4f0ac68.xml (deflated 37%) 2023-11-14T07:55:02.1803921Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdf671b029d36e99.xml (deflated 37%) 2023-11-14T07:55:02.1805951Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7f932e493aebbd60.xml (deflated 37%) 2023-11-14T07:55:02.1807994Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-72f1ad40c275a89a.xml (deflated 44%) 2023-11-14T07:55:02.1810045Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5b0bb5a7ca6f9e7.xml (deflated 44%) 2023-11-14T07:55:02.1812336Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41f63949e57bae30.xml (deflated 44%) 2023-11-14T07:55:02.1814387Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cf98192a0804b86.xml (deflated 44%) 2023-11-14T07:55:02.1816421Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cce43bac22c52341.xml (deflated 44%) 2023-11-14T07:55:02.1818479Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd979df73b5253c0.xml (deflated 44%) 2023-11-14T07:55:02.1820534Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8fcf15850e9d78fa.xml (deflated 43%) 2023-11-14T07:55:02.1822579Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d107f38ab733c5c.xml (deflated 43%) 2023-11-14T07:55:02.1851130Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c26aebe0710774b8.xml (deflated 43%) 2023-11-14T07:55:02.1854913Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd794b600db87c20.xml (deflated 43%) 2023-11-14T07:55:02.1857288Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0c19de9fb961fbb.xml (deflated 43%) 2023-11-14T07:55:02.1859359Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7a0cce4009afe1e.xml (deflated 43%) 2023-11-14T07:55:02.1861424Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-601f638d4881637a.xml (deflated 43%) 2023-11-14T07:55:02.1863415Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-238203092d693ef3.xml (deflated 43%) 2023-11-14T07:55:02.1865427Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e90363274ee401d.xml (deflated 43%) 2023-11-14T07:55:02.1867631Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aeaf7faa85d8a109.xml (deflated 42%) 2023-11-14T07:55:02.1869700Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2632d8ce2e85359.xml (deflated 38%) 2023-11-14T07:55:02.1871695Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4026f1304b3f7d7c.xml (deflated 37%) 2023-11-14T07:55:02.1873845Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-026299f0ad98a575.xml (deflated 43%) 2023-11-14T07:55:02.1875862Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c2c85c577fa0a58.xml (deflated 38%) 2023-11-14T07:55:02.1877864Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70b7e5e4713484c8.xml (deflated 38%) 2023-11-14T07:55:02.1879886Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee01fd19e78fe6c5.xml (deflated 38%) 2023-11-14T07:55:02.1881902Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc1e27bc1ee49f9c.xml (deflated 49%) 2023-11-14T07:55:02.1883995Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c9f3b4ef27774e0.xml (deflated 44%) 2023-11-14T07:55:02.1885998Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de9bffd925f7f136.xml (deflated 44%) 2023-11-14T07:55:02.1888002Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-529fd5f0ad1a9a98.xml (deflated 37%) 2023-11-14T07:55:02.1890020Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef55eaa5dcfdf137.xml (deflated 37%) 2023-11-14T07:55:02.1892358Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e98c438a592e604.xml (deflated 37%) 2023-11-14T07:55:02.1894396Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a77bb6081810a76b.xml (deflated 44%) 2023-11-14T07:55:02.1896432Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27dd25f4c6c02073.xml (deflated 48%) 2023-11-14T07:55:02.1898464Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d60e505cefaeb33.xml (deflated 37%) 2023-11-14T07:55:02.1900519Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64efaa78d95e1ae5.xml (deflated 37%) 2023-11-14T07:55:02.1902530Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9e98bae6284b47f.xml (deflated 44%) 2023-11-14T07:55:02.1904555Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af3beb36423d3ce8.xml (deflated 38%) 2023-11-14T07:55:02.1906601Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed0683e349c1b3b2.xml (deflated 44%) 2023-11-14T07:55:02.1908639Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-598c6d44c601dbe4.xml (deflated 38%) 2023-11-14T07:55:02.1910651Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91db6e2c4a38ed86.xml (deflated 43%) 2023-11-14T07:55:02.1912648Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3b05e3f47700c53.xml (deflated 44%) 2023-11-14T07:55:02.1914660Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-deee3f27f61e8431.xml (deflated 44%) 2023-11-14T07:55:02.1916811Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc1da6e8d2c78b1d.xml (deflated 38%) 2023-11-14T07:55:02.1918855Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6184a1c703567f36.xml (deflated 38%) 2023-11-14T07:55:02.1921023Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1264e54fc33b2ba.xml (deflated 37%) 2023-11-14T07:55:02.1923071Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bcbb2f83f1a117a7.xml (deflated 43%) 2023-11-14T07:55:02.1925132Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f70e4a3ecd4dc928.xml (deflated 43%) 2023-11-14T07:55:02.1927183Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b93ed737e8ad8e2e.xml (deflated 43%) 2023-11-14T07:55:02.1929231Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a0f8cc56b3af1ee5.xml (deflated 44%) 2023-11-14T07:55:02.1931459Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b3665caf979d191.xml (deflated 37%) 2023-11-14T07:55:02.1933521Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4359bca82397f6a0.xml (deflated 38%) 2023-11-14T07:55:02.1935568Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c071e41764aae1c.xml (deflated 37%) 2023-11-14T07:55:02.1937582Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfc3f9ce2e846710.xml (deflated 37%) 2023-11-14T07:55:02.1939631Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7fef1714f4608b5.xml (deflated 37%) 2023-11-14T07:55:02.1941687Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e00a9ffd731ffb5f.xml (deflated 58%) 2023-11-14T07:55:02.1943729Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6235ae3617d7867d.xml (deflated 38%) 2023-11-14T07:55:02.1945756Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a7be02e0b0fa960.xml (deflated 38%) 2023-11-14T07:55:02.1947810Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a0e69574cb8d2b33.xml (deflated 38%) 2023-11-14T07:55:02.1949856Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2520f89f9f7e2c2.xml (deflated 37%) 2023-11-14T07:55:02.1951900Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f0e5b9ac27769e8.xml (deflated 38%) 2023-11-14T07:55:02.1953929Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d63270c296594a2.xml (deflated 37%) 2023-11-14T07:55:02.1955965Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-553fc856711597c7.xml (deflated 37%) 2023-11-14T07:55:02.1958020Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51dda38f57dc3d18.xml (deflated 37%) 2023-11-14T07:55:02.1960060Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-516688b40380c367.xml (deflated 38%) 2023-11-14T07:55:02.1962086Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85343bd221969fbe.xml (deflated 37%) 2023-11-14T07:55:02.1964272Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd774eac5a8feaa5.xml (deflated 43%) 2023-11-14T07:55:02.1966364Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3dd2e159bd780767.xml (deflated 37%) 2023-11-14T07:55:02.1968408Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52ed4b8c07360f99.xml (deflated 37%) 2023-11-14T07:55:02.1970569Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3ee4653e1de46d9.xml (deflated 38%) 2023-11-14T07:55:02.1972848Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-401743db1f3d97ed.xml (deflated 37%) 2023-11-14T07:55:02.1974900Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1a79ef2101dc9eb.xml (deflated 37%) 2023-11-14T07:55:02.1976963Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-272c338699a5f3c8.xml (deflated 37%) 2023-11-14T07:55:02.1978981Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-158cfd91fd83f7d1.xml (deflated 37%) 2023-11-14T07:55:02.1981021Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4853a4b9513986fb.xml (deflated 38%) 2023-11-14T07:55:02.1983110Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9851f0c12f2bb39.xml (deflated 38%) 2023-11-14T07:55:02.1985163Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f383bde7d21d4a4.xml (deflated 37%) 2023-11-14T07:55:02.1987202Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6660793d7edc4aa5.xml (deflated 37%) 2023-11-14T07:55:02.1989249Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6481ac3df207f152.xml (deflated 49%) 2023-11-14T07:55:02.1991295Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ccade17c0ad1406.xml (deflated 37%) 2023-11-14T07:55:02.1993349Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5fef7f172ba59e2f.xml (deflated 38%) 2023-11-14T07:55:02.1995371Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9947295e4787ea7b.xml (deflated 38%) 2023-11-14T07:55:02.1997419Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a65b9d446d069da9.xml (deflated 37%) 2023-11-14T07:55:02.1999465Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa1bade3967b45b0.xml (deflated 38%) 2023-11-14T07:55:02.2001521Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24717d69fddb5810.xml (deflated 37%) 2023-11-14T07:55:02.2003533Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53e02473c9873f23.xml (deflated 43%) 2023-11-14T07:55:02.2005565Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1778b10e357b768d.xml (deflated 43%) 2023-11-14T07:55:02.2007616Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65111be29e517e80.xml (deflated 37%) 2023-11-14T07:55:02.2009649Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18af637c6cd7ce59.xml (deflated 38%) 2023-11-14T07:55:02.2011906Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5195439db97ae2a6.xml (deflated 37%) 2023-11-14T07:55:02.2014074Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b3526a7a947a577.xml (deflated 44%) 2023-11-14T07:55:02.2016141Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-47683f037963c11f.xml (deflated 44%) 2023-11-14T07:55:02.2018174Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24f4f780a40267d5.xml (deflated 44%) 2023-11-14T07:55:02.2020324Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b67d8861a24ad52.xml (deflated 44%) 2023-11-14T07:55:02.2022358Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9996f801b02a5d9a.xml (deflated 38%) 2023-11-14T07:55:02.2024410Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1db435090125da79.xml (deflated 38%) 2023-11-14T07:55:02.2026463Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aedfc718e9bb3f48.xml (deflated 43%) 2023-11-14T07:55:02.2028497Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7167e55127c6f3f9.xml (deflated 43%) 2023-11-14T07:55:02.2030546Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9bd7eac1c9096852.xml (deflated 37%) 2023-11-14T07:55:02.2032577Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16807b82911edc2d.xml (deflated 38%) 2023-11-14T07:55:02.2034618Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4661078ffe114f12.xml (deflated 37%) 2023-11-14T07:55:02.2036630Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fab3478bd2578b04.xml (deflated 37%) 2023-11-14T07:55:02.2038689Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f048e398af62436d.xml (deflated 37%) 2023-11-14T07:55:02.2040725Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8754dd5e3f3d48ee.xml (deflated 43%) 2023-11-14T07:55:02.2042770Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-370953427c7b1a66.xml (deflated 43%) 2023-11-14T07:55:02.2044790Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1005212d239bf9a3.xml (deflated 43%) 2023-11-14T07:55:02.2046817Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-311a253145edfced.xml (deflated 43%) 2023-11-14T07:55:02.2048857Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d122f2d378a602a4.xml (deflated 38%) 2023-11-14T07:55:02.2051043Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e61f5f26bd98746.xml (deflated 38%) 2023-11-14T07:55:02.2053363Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1e29212256c18a6.xml (deflated 43%) 2023-11-14T07:55:02.2055443Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7940fd9da65057a3.xml (deflated 43%) 2023-11-14T07:55:02.2057471Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-637a5e0b29d40277.xml (deflated 44%) 2023-11-14T07:55:02.2059490Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-06e8e09aafb072ae.xml (deflated 46%) 2023-11-14T07:55:02.2061533Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9bf289f30cb43aad.xml (deflated 45%) 2023-11-14T07:55:02.2063708Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70863f14cdfcbed8.xml (deflated 45%) 2023-11-14T07:55:02.2065786Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f14b93d889ae808.xml (deflated 38%) 2023-11-14T07:55:02.2067946Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64845d4ace192d30.xml (deflated 46%) 2023-11-14T07:55:02.2069985Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e724fe67ba8c673c.xml (deflated 45%) 2023-11-14T07:55:02.2072033Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e0aa16a35040ffdc.xml (deflated 43%) 2023-11-14T07:55:02.2074086Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c2e87db9ee5b1be.xml (deflated 37%) 2023-11-14T07:55:02.2076122Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a35d413c240e3fd2.xml (deflated 38%) 2023-11-14T07:55:02.2078166Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a5e1abc3d350c34.xml (deflated 38%) 2023-11-14T07:55:02.2080222Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1458228c85acf512.xml (deflated 37%) 2023-11-14T07:55:02.2082316Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-229914233c08999c.xml (deflated 48%) 2023-11-14T07:55:02.2084363Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-83552a7fbada1332.xml (deflated 39%) 2023-11-14T07:55:02.2086400Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-279aa49e8992c309.xml (deflated 38%) 2023-11-14T07:55:02.2088449Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78237aa3297e16ce.xml (deflated 38%) 2023-11-14T07:55:02.2090486Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d186d87e4f3521ef.xml (deflated 38%) 2023-11-14T07:55:02.2092724Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff7a9e784af0356f.xml (deflated 38%) 2023-11-14T07:55:02.2094766Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f19c3054c3168687.xml (deflated 38%) 2023-11-14T07:55:02.2096802Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a96599953f588167.xml (deflated 37%) 2023-11-14T07:55:02.2098832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-548d534919f3d6bf.xml (deflated 37%) 2023-11-14T07:55:02.2100857Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c772032391a2dd5.xml (deflated 37%) 2023-11-14T07:55:02.2102875Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b92308759d8b197f.xml (deflated 37%) 2023-11-14T07:55:02.2104928Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cf30334ba7fa7df.xml (deflated 37%) 2023-11-14T07:55:02.2106973Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13c485ce590cc5a3.xml (deflated 38%) 2023-11-14T07:55:02.2108995Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-042f3a6a9706a0de.xml (deflated 48%) 2023-11-14T07:55:02.2111212Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6e1515859784405.xml (deflated 43%) 2023-11-14T07:55:02.2113288Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40e3f5b65adfa52b.xml (deflated 45%) 2023-11-14T07:55:02.2115330Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29093aa20c45ec35.xml (deflated 45%) 2023-11-14T07:55:02.2117495Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5c0dab5e1ea3fb3.xml (deflated 46%) 2023-11-14T07:55:02.2119538Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d180521cf6c89e6d.xml (deflated 37%) 2023-11-14T07:55:02.2121585Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-beae3faca1c612a2.xml (deflated 37%) 2023-11-14T07:55:02.2123635Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80a51d410068a727.xml (deflated 37%) 2023-11-14T07:55:02.2125650Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2e9ff8809387d3c.xml (deflated 48%) 2023-11-14T07:55:02.2127685Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ced4072b345efaaf.xml (deflated 48%) 2023-11-14T07:55:02.2129739Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34f572ac64d8c89f.xml (deflated 48%) 2023-11-14T07:55:02.2132299Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3882b1a0f0292dfc.xml (deflated 38%) 2023-11-14T07:55:02.2134325Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64d507d15d2a65ca.xml (deflated 37%) 2023-11-14T07:55:02.2136378Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7ef5599c7ad664e.xml (deflated 37%) 2023-11-14T07:55:02.2138423Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4dcde3674c662f58.xml (deflated 38%) 2023-11-14T07:55:02.2140467Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-271dd108604a1f00.xml (deflated 38%) 2023-11-14T07:55:02.2142489Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a2ff130cfafe26b.xml (deflated 38%) 2023-11-14T07:55:02.2144530Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c33107ae9d2d2988.xml (deflated 49%) 2023-11-14T07:55:02.2146564Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3416b3b854431a5.xml (deflated 49%) 2023-11-14T07:55:02.2148603Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b1ff35e91c703f7.xml (deflated 38%) 2023-11-14T07:55:02.2150629Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bbbd122362e78d39.xml (deflated 38%) 2023-11-14T07:55:02.2152679Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5be3f2ac071c7ed.xml (deflated 38%) 2023-11-14T07:55:02.2154727Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4b27e1459ec2a29.xml (deflated 38%) 2023-11-14T07:55:02.2156772Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b528b774e39c541.xml (deflated 38%) 2023-11-14T07:55:02.2158786Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d2a741459ea03ed.xml (deflated 43%) 2023-11-14T07:55:02.2160970Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-984f9f468863e529.xml (deflated 43%) 2023-11-14T07:55:02.2163052Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a5adcabf070a5da.xml (deflated 43%) 2023-11-14T07:55:02.2165101Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5adc4532ccc4d534.xml (deflated 43%) 2023-11-14T07:55:02.2167255Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39585e0f74088a50.xml (deflated 44%) 2023-11-14T07:55:02.2169292Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecfba24820c3be2c.xml (deflated 44%) 2023-11-14T07:55:02.2171524Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17fe2dc96d53e4bd.xml (deflated 44%) 2023-11-14T07:55:02.2173597Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa8dc6e0bc9a98ac.xml (deflated 45%) 2023-11-14T07:55:02.2175629Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0fd95d2cb436acc0.xml (deflated 44%) 2023-11-14T07:55:02.2177673Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba59018b8b185c34.xml (deflated 44%) 2023-11-14T07:55:02.2179724Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9dbd206c452b5d22.xml (deflated 44%) 2023-11-14T07:55:02.2181757Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45e8467540e497d2.xml (deflated 44%) 2023-11-14T07:55:02.2183819Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55b047e3b8ba3d62.xml (deflated 45%) 2023-11-14T07:55:02.2185872Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5818caf18470b4ce.xml (deflated 44%) 2023-11-14T07:55:02.2187917Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba0d0d927004b9bd.xml (deflated 44%) 2023-11-14T07:55:02.2189965Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1f2181e3a3644679.xml (deflated 38%) 2023-11-14T07:55:02.2191980Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ee3ab418c6cbf8e.xml (deflated 44%) 2023-11-14T07:55:02.2194026Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6555ff01f0ce8aa.xml (deflated 38%) 2023-11-14T07:55:02.2196070Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7eb2bfe146c516c7.xml (deflated 38%) 2023-11-14T07:55:02.2198108Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9523c9c856c30fef.xml (deflated 38%) 2023-11-14T07:55:02.2200123Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c35c3a023729c34.xml (deflated 50%) 2023-11-14T07:55:02.2202169Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c7cebbf1c96b8cc.xml (deflated 42%) 2023-11-14T07:55:02.2204216Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e86a5c60ae2d35ad.xml (deflated 42%) 2023-11-14T07:55:02.2206262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0bab2237f93e4602.xml (deflated 42%) 2023-11-14T07:55:02.2208278Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0fd59325607d7b78.xml (deflated 38%) 2023-11-14T07:55:02.2210451Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eafa8eb31a48d159.xml (deflated 37%) 2023-11-14T07:55:02.2212705Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-198ad3831e8fba4a.xml (deflated 49%) 2023-11-14T07:55:02.2214889Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-999eb58a57f262e7.xml (deflated 37%) 2023-11-14T07:55:02.2216902Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-690970740f8757c0.xml (deflated 37%) 2023-11-14T07:55:02.2218932Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9249a992d698376f.xml (deflated 47%) 2023-11-14T07:55:02.2220972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-edde9a1b1abb0f27.xml (deflated 47%) 2023-11-14T07:55:02.2223026Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f8572bd5a2dc3d39.xml (deflated 37%) 2023-11-14T07:55:02.2225044Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70649c694b62f279.xml (deflated 37%) 2023-11-14T07:55:02.2227074Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e735593f22409d13.xml (deflated 57%) 2023-11-14T07:55:02.2229104Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d5f67a05403d888f.xml (deflated 38%) 2023-11-14T07:55:02.2231145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9346c9bfd9719027.xml (deflated 38%) 2023-11-14T07:55:02.2233158Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f9967d629872fe2.xml (deflated 38%) 2023-11-14T07:55:02.2235207Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26e9c78bb83ec4b0.xml (deflated 37%) 2023-11-14T07:55:02.2237257Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b67b6dba1e1cb637.xml (deflated 37%) 2023-11-14T07:55:02.2239309Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-579cc626e152e1b8.xml (deflated 37%) 2023-11-14T07:55:02.2241321Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10f8ba586e66f69e.xml (deflated 37%) 2023-11-14T07:55:02.2243353Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b97549246a18384.xml (deflated 37%) 2023-11-14T07:55:02.2245382Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-761bd0977b6f9177.xml (deflated 38%) 2023-11-14T07:55:02.2247408Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-46d214fdcf33e2ab.xml (deflated 37%) 2023-11-14T07:55:02.2249454Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c41a3ebb536c6cd7.xml (deflated 37%) 2023-11-14T07:55:02.2251672Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-82840db99649a742.xml (deflated 37%) 2023-11-14T07:55:02.2253719Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1bec5b93bad9fb4f.xml (deflated 38%) 2023-11-14T07:55:02.2255757Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eaedf5db91d3da1d.xml (deflated 38%) 2023-11-14T07:55:02.2257929Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98c0b969925e186a.xml (deflated 38%) 2023-11-14T07:55:02.2259999Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fcb326e441b1f5e3.xml (deflated 37%) 2023-11-14T07:55:02.2262045Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af03272bed452c38.xml (deflated 37%) 2023-11-14T07:55:02.2264198Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af3f9d458a417ab1.xml (deflated 37%) 2023-11-14T07:55:02.2266239Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bb1278c5b707d16.xml (deflated 39%) 2023-11-14T07:55:02.2268277Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7913db839e2d2043.xml (deflated 38%) 2023-11-14T07:55:02.2270317Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95c4895375d38cfb.xml (deflated 38%) 2023-11-14T07:55:02.2272331Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62971e414c7f5b3c.xml (deflated 37%) 2023-11-14T07:55:02.2274361Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3206ef0f8b70b093.xml (deflated 49%) 2023-11-14T07:55:02.2276399Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9cebf142908823b8.xml (deflated 37%) 2023-11-14T07:55:02.2278435Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54bbb2e15b2dc1d6.xml (deflated 38%) 2023-11-14T07:55:02.2280463Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c1d6b50abaa098f.xml (deflated 37%) 2023-11-14T07:55:02.2282520Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2adc3fbd8358aa2.xml (deflated 38%) 2023-11-14T07:55:02.2284613Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9765a2344f8baed7.xml (deflated 37%) 2023-11-14T07:55:02.2286655Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ae1999368c2c025.xml (deflated 37%) 2023-11-14T07:55:02.2288684Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27ad60fc6196102f.xml (deflated 38%) 2023-11-14T07:55:02.2290870Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a826801529dd0d24.xml (deflated 49%) 2023-11-14T07:55:02.2292928Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b8712ae9088fe66.xml (deflated 37%) 2023-11-14T07:55:02.2294971Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b64ab0bfa0462767.xml (deflated 38%) 2023-11-14T07:55:02.2296993Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bce5ef06e85278a7.xml (deflated 37%) 2023-11-14T07:55:02.2299032Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8fc87b3682be3f94.xml (deflated 38%) 2023-11-14T07:55:02.2301085Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48f6a80630d3355d.xml (deflated 49%) 2023-11-14T07:55:02.2303129Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7fd090db15ece15d.xml (deflated 37%) 2023-11-14T07:55:02.2305158Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-adf63e2ea609ed29.xml (deflated 48%) 2023-11-14T07:55:02.2307340Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a314055646aa1049.xml (deflated 45%) 2023-11-14T07:55:02.2309417Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2da69a4dcedb0496.xml (deflated 45%) 2023-11-14T07:55:02.2311465Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebc5e72590c7cc3f.xml (deflated 43%) 2023-11-14T07:55:02.2313630Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-affa8d63d3c02f4b.xml (deflated 43%) 2023-11-14T07:55:02.2315675Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63a4d3ade4c59865.xml (deflated 37%) 2023-11-14T07:55:02.2317714Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-070fb8a1bb399262.xml (deflated 38%) 2023-11-14T07:55:02.2319769Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13b80c786f86fbf8.xml (deflated 37%) 2023-11-14T07:55:02.2321786Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d0273b7a935bc8c.xml (deflated 37%) 2023-11-14T07:55:02.2323830Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b457eaa1c53f207b.xml (deflated 36%) 2023-11-14T07:55:02.2325889Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2fd44d0df796f70.xml (deflated 38%) 2023-11-14T07:55:02.2327930Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-755c6d2ef2b2346c.xml (deflated 38%) 2023-11-14T07:55:02.2330259Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37b28a01f78ea6ac.xml (deflated 49%) 2023-11-14T07:55:02.2332490Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-201874b50b0ce832.xml (deflated 37%) 2023-11-14T07:55:02.2334539Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53bd68d7472e0e04.xml (deflated 45%) 2023-11-14T07:55:02.2336591Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-552e229e61e73363.xml (deflated 43%) 2023-11-14T07:55:02.2338617Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e27c124df389ce64.xml (deflated 38%) 2023-11-14T07:55:02.2340663Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52facfa580e228f6.xml (deflated 38%) 2023-11-14T07:55:02.2342704Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ca1bd400512bc00.xml (deflated 38%) 2023-11-14T07:55:02.2344759Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db8bc0f0441e7f20.xml (deflated 38%) 2023-11-14T07:55:02.2346792Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9f6a4fecaeb2d42.xml (deflated 37%) 2023-11-14T07:55:02.2348848Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e41bf160c2b93631.xml (deflated 38%) 2023-11-14T07:55:02.2350891Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05d7d1400db86789.xml (deflated 41%) 2023-11-14T07:55:02.2352930Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a44b34b8de255f5.xml (deflated 38%) 2023-11-14T07:55:02.2354946Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e7d8a10b5e05e89.xml (deflated 37%) 2023-11-14T07:55:02.2357106Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f5eba8100c25b6f.xml (deflated 38%) 2023-11-14T07:55:02.2359184Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ccbf0374f36a0d47.xml (deflated 37%) 2023-11-14T07:55:02.2361340Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62fbdad43e269946.xml (deflated 37%) 2023-11-14T07:55:02.2363363Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49aa7b1118f75736.xml (deflated 38%) 2023-11-14T07:55:02.2365399Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8aa489b8c36c404d.xml (deflated 37%) 2023-11-14T07:55:02.2367437Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc1c33353d9d6a7b.xml (deflated 49%) 2023-11-14T07:55:02.2369488Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d28d4e33778117c9.xml (deflated 39%) 2023-11-14T07:55:02.2371668Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-024633b822c9af9f.xml (deflated 38%) 2023-11-14T07:55:02.2373721Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a383d4ecef72071.xml (deflated 38%) 2023-11-14T07:55:02.2375760Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8c817e81c9955ab.xml (deflated 38%) 2023-11-14T07:55:02.2377796Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10aace10cbdca394.xml (deflated 38%) 2023-11-14T07:55:02.2379814Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fba84449fc83e11f.xml (deflated 38%) 2023-11-14T07:55:02.2381855Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d779b198a9649e8.xml (deflated 37%) 2023-11-14T07:55:02.2383987Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a7644a0d8b1d59e.xml (deflated 37%) 2023-11-14T07:55:02.2386037Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f81b1a202a956eea.xml (deflated 38%) 2023-11-14T07:55:02.2388061Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3507dfcecf8e562.xml (deflated 38%) 2023-11-14T07:55:02.2390103Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ea29d5aac9b4b0c.xml (deflated 38%) 2023-11-14T07:55:02.2392139Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2424702176952612.xml (deflated 37%) 2023-11-14T07:55:02.2394173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-842e5dca6d355782.xml (deflated 48%) 2023-11-14T07:55:02.2396187Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-017a6b7074b77b5c.xml (deflated 43%) 2023-11-14T07:55:02.2398229Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-892ee7cb0649449b.xml (deflated 45%) 2023-11-14T07:55:02.2400272Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-104aad6bebd02537.xml (deflated 45%) 2023-11-14T07:55:02.2402311Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-586dbc6a385fe779.xml (deflated 45%) 2023-11-14T07:55:02.2404447Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cda1a9dcf6a5c5d5.xml (deflated 37%) 2023-11-14T07:55:02.2406527Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e19c7aa416446e3e.xml (deflated 37%) 2023-11-14T07:55:02.2408568Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e75afb3d9102ed4.xml (deflated 38%) 2023-11-14T07:55:02.2410882Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5dc54c733e28418e.xml (deflated 49%) 2023-11-14T07:55:02.2412923Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3327a4fbbe2c17f2.xml (deflated 48%) 2023-11-14T07:55:02.2414965Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5549cad6bd873d82.xml (deflated 48%) 2023-11-14T07:55:02.2417010Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3049c053134aa6ca.xml (deflated 38%) 2023-11-14T07:55:02.2419049Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a08d9d3c28cd0c6.xml (deflated 37%) 2023-11-14T07:55:02.2421066Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86481b50c015072c.xml (deflated 37%) 2023-11-14T07:55:02.2423110Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cdded053a67df276.xml (deflated 38%) 2023-11-14T07:55:02.2425150Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6c65d3737fedc17.xml (deflated 38%) 2023-11-14T07:55:02.2427189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-141d5aba0c9c2fcc.xml (deflated 38%) 2023-11-14T07:55:02.2429211Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-251b025120e5d60f.xml (deflated 49%) 2023-11-14T07:55:02.2431243Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a17782fc702ed545.xml (deflated 49%) 2023-11-14T07:55:02.2433288Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be9823ae1fedb83b.xml (deflated 38%) 2023-11-14T07:55:02.2435337Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-edb687c8f349b416.xml (deflated 38%) 2023-11-14T07:55:02.2437356Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d82593f922e2bc1e.xml (deflated 38%) 2023-11-14T07:55:02.2439391Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59a28d1632b13927.xml (deflated 38%) 2023-11-14T07:55:02.2441431Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7343c50dd688c068.xml (deflated 38%) 2023-11-14T07:55:02.2443461Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-935251387091f279.xml (deflated 43%) 2023-11-14T07:55:02.2445472Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-536e7b63a8fd9678.xml (deflated 44%) 2023-11-14T07:55:02.2447519Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f30c5a01d4a3eb9.xml (deflated 43%) 2023-11-14T07:55:02.2449557Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6148d6b29159c43b.xml (deflated 43%) 2023-11-14T07:55:02.2451751Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7bf0a99728f0dcd.xml (deflated 44%) 2023-11-14T07:55:02.2453900Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45530d7baab8f422.xml (deflated 44%) 2023-11-14T07:55:02.2455972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c51aa6442a235226.xml (deflated 44%) 2023-11-14T07:55:02.2458013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d010af937d8b497d.xml (deflated 45%) 2023-11-14T07:55:02.2460163Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b41d07e5d5d83a94.xml (deflated 44%) 2023-11-14T07:55:02.2462184Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b6b3d720a28a4ef.xml (deflated 44%) 2023-11-14T07:55:02.2464234Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f8724b3cdd4342f4.xml (deflated 44%) 2023-11-14T07:55:02.2466282Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-50e5406f965d69bb.xml (deflated 44%) 2023-11-14T07:55:02.2468324Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d66b88e9a464dcbe.xml (deflated 45%) 2023-11-14T07:55:02.2470352Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-591ec9a00a879d9a.xml (deflated 44%) 2023-11-14T07:55:02.2472403Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0dfa5b2e7cbb3136.xml (deflated 44%) 2023-11-14T07:55:02.2474452Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-434b7c9affbe0bf5.xml (deflated 38%) 2023-11-14T07:55:02.2476476Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a51a9bafabec7f04.xml (deflated 44%) 2023-11-14T07:55:02.2478521Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e19c1128a79a7a57.xml (deflated 38%) 2023-11-14T07:55:02.2480566Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-77dcd0e6cebecf47.xml (deflated 37%) 2023-11-14T07:55:02.2482618Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4b01a3054bbccab.xml (deflated 37%) 2023-11-14T07:55:02.2484687Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f8608601f2e99cb5.xml (deflated 37%) 2023-11-14T07:55:02.2486724Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2b479518a2bf8b2.xml (deflated 42%) 2023-11-14T07:55:02.2488760Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4cfc93e30c499915.xml (deflated 42%) 2023-11-14T07:55:02.2490964Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-83acab29be483e58.xml (deflated 42%) 2023-11-14T07:55:02.2493005Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-952ccf72c94e6c6e.xml (deflated 38%) 2023-11-14T07:55:02.2495057Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0967ca7fbbfca36d.xml (deflated 37%) 2023-11-14T07:55:02.2497104Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-549a713e0c0b3af4.xml (deflated 49%) 2023-11-14T07:55:02.2499145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2990d730cbf621d.xml (deflated 38%) 2023-11-14T07:55:02.2501157Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3fe91d03b590d23d.xml (deflated 37%) 2023-11-14T07:55:02.2503304Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-31cb404a6f93fe9a.xml (deflated 47%) 2023-11-14T07:55:02.2505385Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db01ccaa3fb62b35.xml (deflated 47%) 2023-11-14T07:55:02.2507556Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8afcdc34e6c6ed2b.xml (deflated 37%) 2023-11-14T07:55:02.2509588Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e1d6007a37cff22.xml (deflated 37%) 2023-11-14T07:55:02.2511619Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-176699a88f0e5ccf.xml (deflated 57%) 2023-11-14T07:55:02.2513654Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3afb4c007954429e.xml (deflated 37%) 2023-11-14T07:55:02.2515707Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4cc0a0cc63dbb95.xml (deflated 37%) 2023-11-14T07:55:02.2517734Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-99ae405f5ba21c6b.xml (deflated 38%) 2023-11-14T07:55:02.2519775Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-035ac2cc72f2ace6.xml (deflated 37%) 2023-11-14T07:55:02.2521820Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a708455ef0490a4a.xml (deflated 37%) 2023-11-14T07:55:02.2523857Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3daaef232537fa8.xml (deflated 37%) 2023-11-14T07:55:02.2525879Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1695b690c3fc1537.xml (deflated 37%) 2023-11-14T07:55:02.2527913Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c75ab19b2601f55.xml (deflated 37%) 2023-11-14T07:55:02.2529962Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4b8aacab8009fc3.xml (deflated 37%) 2023-11-14T07:55:02.2532196Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65b2d637cfc19a61.xml (deflated 37%) 2023-11-14T07:55:02.2534227Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-367805d5c00f767a.xml (deflated 37%) 2023-11-14T07:55:02.2536267Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c6342525ba38fc91.xml (deflated 37%) 2023-11-14T07:55:02.2538307Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f7ddfe852719190.xml (deflated 37%) 2023-11-14T07:55:02.2540360Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9c438ade4e48951.xml (deflated 38%) 2023-11-14T07:55:02.2542383Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8bd156e64e83011.xml (deflated 37%) 2023-11-14T07:55:02.2544433Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd54919f0d4d05a3.xml (deflated 38%) 2023-11-14T07:55:02.2546472Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a4c015c69ec445d.xml (deflated 37%) 2023-11-14T07:55:02.2548512Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92918d80a576a882.xml (deflated 38%) 2023-11-14T07:55:02.2550529Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aea147cb477872c2.xml (deflated 38%) 2023-11-14T07:55:02.2552674Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c26cb4a26cee74b9.xml (deflated 37%) 2023-11-14T07:55:02.2554747Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4692ea8b5ef263e.xml (deflated 37%) 2023-11-14T07:55:02.2556905Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-285121d6a8f16503.xml (deflated 37%) 2023-11-14T07:55:02.2558918Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ed90644eb46dfd5.xml (deflated 49%) 2023-11-14T07:55:02.2560960Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8228f0eb746f53b7.xml (deflated 37%) 2023-11-14T07:55:02.2563009Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb586f62fbdb6cc0.xml (deflated 38%) 2023-11-14T07:55:02.2565056Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-597b8ebf4ac65d73.xml (deflated 37%) 2023-11-14T07:55:02.2567083Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb19882a252fdba4.xml (deflated 37%) 2023-11-14T07:55:02.2569135Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c027ed6dcda7a090.xml (deflated 38%) 2023-11-14T07:55:02.2571341Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2632fdd2846e06b.xml (deflated 37%) 2023-11-14T07:55:02.2573386Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-427996318cd5fe55.xml (deflated 38%) 2023-11-14T07:55:02.2575408Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dad6596789a5ca08.xml (deflated 49%) 2023-11-14T07:55:02.2577451Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e2e60a2d44d3a2ed.xml (deflated 37%) 2023-11-14T07:55:02.2579496Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20ad15f734ce7921.xml (deflated 38%) 2023-11-14T07:55:02.2581543Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48a4c483eef4d6c5.xml (deflated 37%) 2023-11-14T07:55:02.2583607Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-601cfffaeb5fdb0f.xml (deflated 38%) 2023-11-14T07:55:02.2585650Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14951a87446a38b8.xml (deflated 49%) 2023-11-14T07:55:02.2587694Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ad38e5b5ecfcf05f.xml (deflated 37%) 2023-11-14T07:55:02.2589744Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fb00b9cb39e2810.xml (deflated 48%) 2023-11-14T07:55:02.2591765Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e39a0a3472775ffe.xml (deflated 45%) 2023-11-14T07:55:02.2593800Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60e305459126bea1.xml (deflated 45%) 2023-11-14T07:55:02.2595847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfecc5f2bedfacd5.xml (deflated 43%) 2023-11-14T07:55:02.2597898Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f69c2193d7d74be9.xml (deflated 43%) 2023-11-14T07:55:02.2600048Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b143c42dd1f5ebc7.xml (deflated 37%) 2023-11-14T07:55:02.2602117Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66683c9fc18d1108.xml (deflated 38%) 2023-11-14T07:55:02.2604149Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-836298079a58d421.xml (deflated 37%) 2023-11-14T07:55:02.2606343Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7f9037936b18d48.xml (deflated 37%) 2023-11-14T07:55:02.2608353Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07d12e1d520016c3.xml (deflated 36%) 2023-11-14T07:55:02.2610396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5eefc814f347db2f.xml (deflated 37%) 2023-11-14T07:55:02.2612618Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e037ec04e6ee48cb.xml (deflated 38%) 2023-11-14T07:55:02.2614670Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4eadab9ede7594e5.xml (deflated 49%) 2023-11-14T07:55:02.2616701Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac00c9e8ac06fc06.xml (deflated 38%) 2023-11-14T07:55:02.2618752Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6389d80ac8df0ece.xml (deflated 45%) 2023-11-14T07:55:02.2620795Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-801e76d65cb6ee8d.xml (deflated 43%) 2023-11-14T07:55:02.2622835Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d705ada2747a8462.xml (deflated 38%) 2023-11-14T07:55:02.2624853Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07494b35ccae5618.xml (deflated 38%) 2023-11-14T07:55:02.2626901Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ebb94acadaefd2a.xml (deflated 38%) 2023-11-14T07:55:02.2628953Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-68abccc6a8ac16fc.xml (deflated 38%) 2023-11-14T07:55:02.2631003Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8e2ada62eaa6668.xml (deflated 37%) 2023-11-14T07:55:02.2633038Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae73e7b9bdab44c2.xml (deflated 37%) 2023-11-14T07:55:02.2635087Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6ca8ee13b3d43c19.xml (deflated 41%) 2023-11-14T07:55:02.2637133Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30e9fbe8832cf4ae.xml (deflated 38%) 2023-11-14T07:55:02.2639176Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b9f8609684c21e5.xml (deflated 38%) 2023-11-14T07:55:02.2641206Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89c887027be266bf.xml (deflated 37%) 2023-11-14T07:55:02.2643246Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3054bd4efe0f924f.xml (deflated 37%) 2023-11-14T07:55:02.2645293Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e0d00c16fd4a3db.xml (deflated 37%) 2023-11-14T07:55:02.2647339Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c82bcadb378d4f18.xml (deflated 38%) 2023-11-14T07:55:02.2649485Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c277521bf60e08bb.xml (deflated 37%) 2023-11-14T07:55:02.2651729Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9efd2f2d41dc4c75.xml (deflated 49%) 2023-11-14T07:55:02.2653925Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fb3b3aa6d10119b6.xml (deflated 39%) 2023-11-14T07:55:02.2655964Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9995933d20e8dd3.xml (deflated 38%) 2023-11-14T07:55:02.2657976Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-824eb2f662735eb1.xml (deflated 38%) 2023-11-14T07:55:02.2660001Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-251d1c5a3867800e.xml (deflated 37%) 2023-11-14T07:55:02.2662034Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36167b66c6048847.xml (deflated 38%) 2023-11-14T07:55:02.2664055Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34238723173fdc59.xml (deflated 38%) 2023-11-14T07:55:02.2666072Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b721d0e9fc0271a6.xml (deflated 44%) 2023-11-14T07:55:02.2668095Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-548bedd17d61563a.xml (deflated 44%) 2023-11-14T07:55:02.2670123Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3939913068963bc6.xml (deflated 44%) 2023-11-14T07:55:02.2672125Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6fec7e751a09b2b.xml (deflated 38%) 2023-11-14T07:55:02.2674168Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ac7803ab496999b.xml (deflated 45%) 2023-11-14T07:55:02.2676198Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ac7b47963cd526f.xml (deflated 45%) 2023-11-14T07:55:02.2678237Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a87c8b7c02d77011.xml (deflated 46%) 2023-11-14T07:55:02.2680243Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-062f354a80c9003d.xml (deflated 43%) 2023-11-14T07:55:02.2682316Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-612594180c7a334f.xml (deflated 45%) 2023-11-14T07:55:02.2684378Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e15f59b8615e984.xml (deflated 45%) 2023-11-14T07:55:02.2686405Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-659f02b19ea47eee.xml (deflated 46%) 2023-11-14T07:55:02.2688417Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1850970a7d9f905b.xml (deflated 38%) 2023-11-14T07:55:02.2690448Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5f5e8cd02421831.xml (deflated 44%) 2023-11-14T07:55:02.2693038Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ebd9165e16e197d.xml (deflated 44%) 2023-11-14T07:55:02.2695082Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6abb1bb5b1384f4a.xml (deflated 44%) 2023-11-14T07:55:02.2697095Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7bd70dacad80222.xml (deflated 44%) 2023-11-14T07:55:02.2699263Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3f3b69ec8160376.xml (deflated 44%) 2023-11-14T07:55:02.2701333Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d0691b5cc05adb7.xml (deflated 44%) 2023-11-14T07:55:02.2703495Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9d5dab2ec8bb894.xml (deflated 37%) 2023-11-14T07:55:02.2705501Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0652134f806cd76d.xml (deflated 37%) 2023-11-14T07:55:02.2707557Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7dec3dd8567cb251.xml (deflated 38%) 2023-11-14T07:55:02.2709665Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b0137aff45ab4a9.xml (deflated 38%) 2023-11-14T07:55:02.2711761Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e242eef12462a35.xml (deflated 38%) 2023-11-14T07:55:02.2713783Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-636b5a49a8116603.xml (deflated 49%) 2023-11-14T07:55:02.2715810Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55e11960bd7b413b.xml (deflated 48%) 2023-11-14T07:55:02.2717823Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66611a977788ea0e.xml (deflated 38%) 2023-11-14T07:55:02.2719835Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18251fa92968b03b.xml (deflated 37%) 2023-11-14T07:55:02.2721856Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7f3c6a866847600.xml (deflated 37%) 2023-11-14T07:55:02.2723886Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-82d51c3b0a42ebe2.xml (deflated 44%) 2023-11-14T07:55:02.2725888Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f31500b27f604fb6.xml (deflated 44%) 2023-11-14T07:55:02.2727918Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0b38e067c3a046c.xml (deflated 43%) 2023-11-14T07:55:02.2729937Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4134150f7599783.xml (deflated 43%) 2023-11-14T07:55:02.2732252Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-67b472c223c06aa6.xml (deflated 43%) 2023-11-14T07:55:02.2734259Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3cc031cfd9988cef.xml (deflated 43%) 2023-11-14T07:55:02.2736300Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4b221e1e2ef3476.xml (deflated 45%) 2023-11-14T07:55:02.2738331Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca29b72246906cad.xml (deflated 44%) 2023-11-14T07:55:02.2740373Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3151c65c938afa9a.xml (deflated 44%) 2023-11-14T07:55:02.2742386Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b042eb72f8d0d934.xml (deflated 45%) 2023-11-14T07:55:02.2744416Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-123554d2eadf8a9c.xml (deflated 44%) 2023-11-14T07:55:02.2746557Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c6ce8193f12614c.xml (deflated 45%) 2023-11-14T07:55:02.2748612Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-870bafb679495005.xml (deflated 44%) 2023-11-14T07:55:02.2750612Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c399cafc4c7b443.xml (deflated 44%) 2023-11-14T07:55:02.2752774Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdad373f46edf1b2.xml (deflated 45%) 2023-11-14T07:55:02.2754810Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9693950f331f0f8.xml (deflated 44%) 2023-11-14T07:55:02.2756835Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da438cc39f1aedac.xml (deflated 44%) 2023-11-14T07:55:02.2758854Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65666c35b2e3b527.xml (deflated 38%) 2023-11-14T07:55:02.2760879Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-363acba8dd921d5e.xml (deflated 44%) 2023-11-14T07:55:02.2762918Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9aaebe9b5bfa90ac.xml (deflated 38%) 2023-11-14T07:55:02.2764957Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-825f40db14c5ef33.xml (deflated 43%) 2023-11-14T07:55:02.2766962Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-203fbfb483a08cdd.xml (deflated 44%) 2023-11-14T07:55:02.2769000Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bcbb976010d37e2.xml (deflated 44%) 2023-11-14T07:55:02.2771274Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a95c79ebcc86710f.xml (deflated 42%) 2023-11-14T07:55:02.2773326Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-233d4b486a3bcbff.xml (deflated 42%) 2023-11-14T07:55:02.2775332Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-33c671fc704ea161.xml (deflated 42%) 2023-11-14T07:55:02.2777372Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ad2f3b5b66e86a6.xml (deflated 38%) 2023-11-14T07:55:02.2779396Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-960de975803b89cb.xml (deflated 37%) 2023-11-14T07:55:02.2781426Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c49b4ca7017676f.xml (deflated 49%) 2023-11-14T07:55:02.2783483Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-909ac17954ca33b3.xml (deflated 38%) 2023-11-14T07:55:02.2785521Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8bdad9fe71f8b49d.xml (deflated 94%) 2023-11-14T07:55:02.2787547Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-392198106c388806.xml (deflated 95%) 2023-11-14T07:55:02.2789572Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fad1e8098225ecaa.xml (deflated 47%) 2023-11-14T07:55:02.2791585Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-957bc4d267df0b1c.xml (deflated 47%) 2023-11-14T07:55:02.2793617Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b180c6f34341c07e.xml (deflated 37%) 2023-11-14T07:55:02.2795788Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b2c1fbe2a5ee19a.xml (deflated 37%) 2023-11-14T07:55:02.2797853Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1bac0f240ea01924.xml (deflated 57%) 2023-11-14T07:55:02.2799862Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fea7b0c4be027a8e.xml (deflated 38%) 2023-11-14T07:55:02.2802022Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e6975418dcfc618.xml (deflated 37%) 2023-11-14T07:55:02.2804047Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-223cd7ec5fa2cfe5.xml (deflated 38%) 2023-11-14T07:55:02.2806087Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a882cd33429420c4.xml (deflated 37%) 2023-11-14T07:55:02.2808097Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c13b3ef4915302e1.xml (deflated 37%) 2023-11-14T07:55:02.2810122Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1ba40569b19dbdd.xml (deflated 37%) 2023-11-14T07:55:02.2812336Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71585e8e03013eee.xml (deflated 37%) 2023-11-14T07:55:02.2814369Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fbf4f05b0440cddd.xml (deflated 37%) 2023-11-14T07:55:02.2816379Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48877715bcf9f5ed.xml (deflated 38%) 2023-11-14T07:55:02.2818409Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c34d29cb2e8604d7.xml (deflated 37%) 2023-11-14T07:55:02.2820445Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-feb86165052942eb.xml (deflated 37%) 2023-11-14T07:55:02.2822471Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d71b8154e337ced9.xml (deflated 37%) 2023-11-14T07:55:02.2824477Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f3059cefd2e68e4.xml (deflated 38%) 2023-11-14T07:55:02.2826517Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34fd237fdc95e7f2.xml (deflated 38%) 2023-11-14T07:55:02.2828540Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15ac746e6903e41c.xml (deflated 38%) 2023-11-14T07:55:02.2830555Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70cabe0e1683c323.xml (deflated 38%) 2023-11-14T07:55:02.2832561Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2a80c8207521c2d.xml (deflated 37%) 2023-11-14T07:55:02.2834587Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecc7c45f61279a66.xml (deflated 38%) 2023-11-14T07:55:02.2836649Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26677226395f4519.xml (deflated 39%) 2023-11-14T07:55:02.2838661Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd5aa89806a6e919.xml (deflated 38%) 2023-11-14T07:55:02.2840722Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-357e4ebe59b1b54f.xml (deflated 93%) 2023-11-14T07:55:02.2842710Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c3e54f9d9264c85.xml (deflated 94%) 2023-11-14T07:55:02.2844822Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e38ececbc8461d34.xml (deflated 38%) 2023-11-14T07:55:02.2846859Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e51b7760cd8b58e.xml (deflated 49%) 2023-11-14T07:55:02.2848855Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07d5c30aa0d94c3e.xml (deflated 37%) 2023-11-14T07:55:02.2851145Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e7b1edb63b95024.xml (deflated 38%) 2023-11-14T07:55:02.2853210Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ff41484c374238d.xml (deflated 37%) 2023-11-14T07:55:02.2855256Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-928dcf467aaabf46.xml (deflated 37%) 2023-11-14T07:55:02.2857293Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3445391f109a2985.xml (deflated 37%) 2023-11-14T07:55:02.2859300Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd990106e5ea4f4c.xml (deflated 37%) 2023-11-14T07:55:02.2861296Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a6db6a55c1eb027.xml (deflated 43%) 2023-11-14T07:55:02.2863312Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20f9b8518054c9ae.xml (deflated 43%) 2023-11-14T07:55:02.2865311Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-68e37e40a3795350.xml (deflated 43%) 2023-11-14T07:55:02.2867303Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6b62e238ead2e1c.xml (deflated 38%) 2023-11-14T07:55:02.2869300Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ece7483ad22102e.xml (deflated 37%) 2023-11-14T07:55:02.2871300Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c6fd4cfb1405d80.xml (deflated 38%) 2023-11-14T07:55:02.2873310Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4979ea01a9c11f0.xml (deflated 44%) 2023-11-14T07:55:02.2875322Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3cb4ddb8310cca42.xml (deflated 44%) 2023-11-14T07:55:02.2877305Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-77919cd5b3a682ab.xml (deflated 44%) 2023-11-14T07:55:02.2879305Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa61e37fde92b5ea.xml (deflated 45%) 2023-11-14T07:55:02.2881316Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f01c9b4c3e413a8.xml (deflated 45%) 2023-11-14T07:55:02.2883359Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f96bef79607a977e.xml (deflated 43%) 2023-11-14T07:55:02.2885347Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a546b61e64abc73.xml (deflated 43%) 2023-11-14T07:55:02.2887350Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9f20b49cd006029.xml (deflated 37%) 2023-11-14T07:55:02.2889360Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-846941ae5d294aea.xml (deflated 37%) 2023-11-14T07:55:02.2891575Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4f564cba4b03de8.xml (deflated 37%) 2023-11-14T07:55:02.2893691Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6b1c498128c165d.xml (deflated 37%) 2023-11-14T07:55:02.2895718Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce7d64a59f351fc6.xml (deflated 37%) 2023-11-14T07:55:02.2897843Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4c59b48a3700c63.xml (deflated 37%) 2023-11-14T07:55:02.2899855Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce0c33e2dabc122b.xml (deflated 38%) 2023-11-14T07:55:02.2901841Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86d5827f6bbebbdc.xml (deflated 49%) 2023-11-14T07:55:02.2903844Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-194bd096aeeecc7b.xml (deflated 37%) 2023-11-14T07:55:02.2905858Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e5aa35540ffead0.xml (deflated 45%) 2023-11-14T07:55:02.2907849Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0906156a0e59ab14.xml (deflated 43%) 2023-11-14T07:55:02.2909830Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fb8a1365b69d744.xml (deflated 37%) 2023-11-14T07:55:02.2911824Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b40bc8933902d6.xml (deflated 43%) 2023-11-14T07:55:02.2913817Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d382e3a8df8128e.xml (deflated 44%) 2023-11-14T07:55:02.2915811Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9d13556bd5f71e8.xml (deflated 38%) 2023-11-14T07:55:02.2917792Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76d9f49a1d6e9622.xml (deflated 45%) 2023-11-14T07:55:02.2919789Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b80533a64bca84a.xml (deflated 38%) 2023-11-14T07:55:02.2921789Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c9785bf2006c466.xml (deflated 41%) 2023-11-14T07:55:02.2923781Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90ac7a0c14864b98.xml (deflated 38%) 2023-11-14T07:55:02.2925750Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-058c4e866f2e1040.xml (deflated 37%) 2023-11-14T07:55:02.2927742Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8547f02cc96e59c8.xml (deflated 37%) 2023-11-14T07:55:02.2929745Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e362f7bbb37cd63.xml (deflated 45%) 2023-11-14T07:55:02.2931984Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2698b952ff59b0bd.xml (deflated 45%) 2023-11-14T07:55:02.2933973Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb864753fae87bf2.xml (deflated 38%) 2023-11-14T07:55:02.2935974Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d3ca16e27fede0d.xml (deflated 37%) 2023-11-14T07:55:02.2937975Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae9633216f6f3e2d.xml (deflated 48%) 2023-11-14T07:55:02.2940096Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_optim_state/distributed.fsdp.test_fsdp_optim_state-d06b512d153724bb.xml (deflated 93%) 2023-11-14T07:55:02.2942342Z adding: test/test-reports/python-pytest/distributed.test_multi_threaded_pg/distributed.test_multi_threaded_pg-cc6c4d75231d6ac3.xml (deflated 83%) 2023-11-14T07:55:02.2944353Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a24307acdd6ef734.xml (deflated 38%) 2023-11-14T07:55:02.2946384Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-aec6b13a3fbb3f49.xml (deflated 39%) 2023-11-14T07:55:02.2948269Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a503d4e51986078b.xml (deflated 44%) 2023-11-14T07:55:02.2950131Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-cfae4e2ae5b60b3e.xml (deflated 37%) 2023-11-14T07:55:02.2952007Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5a99ee664ceea590.xml (deflated 37%) 2023-11-14T07:55:02.2953887Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-42c3885ae0c4f741.xml (deflated 37%) 2023-11-14T07:55:02.2955742Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-af14b341c6f65dd6.xml (deflated 36%) 2023-11-14T07:55:02.2957624Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-49e76f3fc2c2f0bd.xml (deflated 37%) 2023-11-14T07:55:02.2959504Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fa4acc3c90690184.xml (deflated 36%) 2023-11-14T07:55:02.2961371Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8eabe42652cd99b1.xml (deflated 36%) 2023-11-14T07:55:02.2963220Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-af64a7133fea861a.xml (deflated 37%) 2023-11-14T07:55:02.2965098Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a20edb2d2ee9da22.xml (deflated 37%) 2023-11-14T07:55:02.2966972Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ba3109db59a71514.xml (deflated 37%) 2023-11-14T07:55:02.2968837Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a2a5e5c899fedffd.xml (deflated 37%) 2023-11-14T07:55:02.2970911Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ea729d811161eacc.xml (deflated 36%) 2023-11-14T07:55:02.2972801Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6415f129b8472cea.xml (deflated 37%) 2023-11-14T07:55:02.2974670Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7adc53dbb4cdb7bb.xml (deflated 37%) 2023-11-14T07:55:02.2976540Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a3d157ff66eae41d.xml (deflated 38%) 2023-11-14T07:55:02.2978461Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-475a7d0ce41bdae1.xml (deflated 38%) 2023-11-14T07:55:02.2980335Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1d2e895d5d259ddf.xml (deflated 50%) 2023-11-14T07:55:02.2982340Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5a3b833cd0a84187.xml (deflated 37%) 2023-11-14T07:55:02.2984216Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-591b1e3048be605a.xml (deflated 37%) 2023-11-14T07:55:02.2986079Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5e479acd03608037.xml (deflated 37%) 2023-11-14T07:55:02.2987942Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0761aadbb5f81138.xml (deflated 36%) 2023-11-14T07:55:02.2989917Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-232d402bf0830376.xml (deflated 37%) 2023-11-14T07:55:02.2991790Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8fd087c56440d70d.xml (deflated 37%) 2023-11-14T07:55:02.2993650Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-38865aa6aaa4b018.xml (deflated 37%) 2023-11-14T07:55:02.2995680Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-395a31dd1484296d.xml (deflated 37%) 2023-11-14T07:55:02.2997536Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-97e1e0a5f42b2c46.xml (deflated 37%) 2023-11-14T07:55:02.2999372Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-77496407ec441f62.xml (deflated 37%) 2023-11-14T07:55:02.3001241Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1ee0b6410d161b05.xml (deflated 38%) 2023-11-14T07:55:02.3003105Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-79e3bfc127e96293.xml (deflated 38%) 2023-11-14T07:55:02.3004964Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c087d0daa52e0be9.xml (deflated 38%) 2023-11-14T07:55:02.3006816Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c24750e4747dfc46.xml (deflated 38%) 2023-11-14T07:55:02.3008675Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bfa1d2559518ec94.xml (deflated 38%) 2023-11-14T07:55:02.3010545Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-cfef29f2b062b8e7.xml (deflated 38%) 2023-11-14T07:55:02.3012608Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-849af6af64f7f5a1.xml (deflated 38%) 2023-11-14T07:55:02.3014466Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4e75a2c5c15ddc3d.xml (deflated 38%) 2023-11-14T07:55:02.3016322Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9f9856508896d111.xml (deflated 39%) 2023-11-14T07:55:02.3018191Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d958aa340697b748.xml (deflated 39%) 2023-11-14T07:55:02.3020035Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a0e74f332f6534d8.xml (deflated 39%) 2023-11-14T07:55:02.3021891Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ce46c7f6532642b7.xml (deflated 38%) 2023-11-14T07:55:02.3023747Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5ee10c8336b5826e.xml (deflated 37%) 2023-11-14T07:55:02.3025607Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b3ee6a9633716516.xml (deflated 37%) 2023-11-14T07:55:02.3027448Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7f69c0b1e41bff09.xml (deflated 37%) 2023-11-14T07:55:02.3029315Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ef06e0f30d90e389.xml (deflated 38%) 2023-11-14T07:55:02.3031192Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f641b8dc0002e2bd.xml (deflated 37%) 2023-11-14T07:55:02.3033051Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-450eb32172c27fb6.xml (deflated 37%) 2023-11-14T07:55:02.3034895Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-594d44bde116521a.xml (deflated 37%) 2023-11-14T07:55:02.3036864Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4c521417813e8d02.xml (deflated 37%) 2023-11-14T07:55:02.3038753Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e9b2d44079782b32.xml (deflated 38%) 2023-11-14T07:55:02.3040610Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-751f2f26587d6736.xml (deflated 38%) 2023-11-14T07:55:02.3042602Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0cef4948c2593f4f.xml (deflated 37%) 2023-11-14T07:55:02.3044461Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8d8fc98d4f2aafbf.xml (deflated 38%) 2023-11-14T07:55:02.3046324Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2c0668bc89f744ae.xml (deflated 39%) 2023-11-14T07:55:02.3048188Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e6c07f1b86b49767.xml (deflated 38%) 2023-11-14T07:55:02.3050038Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-55bcfb7d921cb019.xml (deflated 39%) 2023-11-14T07:55:02.3052082Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-89460cfe9599623d.xml (deflated 38%) 2023-11-14T07:55:02.3053954Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7e28e0407aaca38a.xml (deflated 38%) 2023-11-14T07:55:02.3055816Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ca1181689395ede8.xml (deflated 49%) 2023-11-14T07:55:02.3057653Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-71d59621115a3049.xml (deflated 38%) 2023-11-14T07:55:02.3059513Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-431a3695f981cc30.xml (deflated 38%) 2023-11-14T07:55:02.3061379Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8e136b582cdd2aa7.xml (deflated 38%) 2023-11-14T07:55:02.3063230Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-74105c0d3e65aea1.xml (deflated 38%) 2023-11-14T07:55:02.3065098Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5bd312007c432a06.xml (deflated 38%) 2023-11-14T07:55:02.3066964Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d89ff188097db619.xml (deflated 39%) 2023-11-14T07:55:02.3068821Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-511170c92b8b72df.xml (deflated 38%) 2023-11-14T07:55:02.3070652Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a76778f09eaf8068.xml (deflated 37%) 2023-11-14T07:55:02.3072516Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9317915993cfecca.xml (deflated 37%) 2023-11-14T07:55:02.3074385Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3196fa69d44d9729.xml (deflated 37%) 2023-11-14T07:55:02.3076246Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8b2be78df1591f9e.xml (deflated 37%) 2023-11-14T07:55:02.3078100Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b4248ad8299a5256.xml (deflated 37%) 2023-11-14T07:55:02.3079955Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7375612222afada1.xml (deflated 38%) 2023-11-14T07:55:02.3081813Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8ad2806fe76d2f22.xml (deflated 38%) 2023-11-14T07:55:02.3083729Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0ad902121e414e1d.xml (deflated 38%) 2023-11-14T07:55:02.3085694Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2d07114cb6feeb8c.xml (deflated 38%) 2023-11-14T07:55:02.3087589Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fe64203edef627f4.xml (deflated 38%) 2023-11-14T07:55:02.3089581Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3fd57e2c02bb150a.xml (deflated 50%) 2023-11-14T07:55:02.3091622Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-06fb67ca26931723.xml (deflated 38%) 2023-11-14T07:55:02.3093469Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0d8371122363a820.xml (deflated 39%) 2023-11-14T07:55:02.3095327Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-09a6046e19fe2ff0.xml (deflated 39%) 2023-11-14T07:55:02.3097194Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-186f66e4aa43cdc4.xml (deflated 38%) 2023-11-14T07:55:02.3099036Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-da3cf43814863d57.xml (deflated 38%) 2023-11-14T07:55:02.3100900Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4b2baaaac4a0acea.xml (deflated 50%) 2023-11-14T07:55:02.3102772Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d0f7dc0133a7a239.xml (deflated 50%) 2023-11-14T07:55:02.3104633Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d53082c646f58f2f.xml (deflated 38%) 2023-11-14T07:55:02.3106475Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e1eec527a23b5288.xml (deflated 50%) 2023-11-14T07:55:02.3108337Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4ad0be56ffc022bd.xml (deflated 39%) 2023-11-14T07:55:02.3110219Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a55488ceb9b38855.xml (deflated 39%) 2023-11-14T07:55:02.3112080Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bf344b3cd0205e34.xml (deflated 38%) 2023-11-14T07:55:02.3113933Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9383fafc0e233402.xml (deflated 38%) 2023-11-14T07:55:02.3115789Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-39c2bb74181148b1.xml (deflated 39%) 2023-11-14T07:55:02.3117653Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f8d05af558e0867c.xml (deflated 39%) 2023-11-14T07:55:02.3119522Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-420fb32e470bbbee.xml (deflated 38%) 2023-11-14T07:55:02.3121371Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f3319026bbde6c45.xml (deflated 37%) 2023-11-14T07:55:02.3123235Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2ddfae0acc4277fe.xml (deflated 38%) 2023-11-14T07:55:02.3125102Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bde27d8adf25523f.xml (deflated 38%) 2023-11-14T07:55:02.3126984Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-acf2f47665ad94e8.xml (deflated 37%) 2023-11-14T07:55:02.3128830Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-93878e75b8bf25b2.xml (deflated 37%) 2023-11-14T07:55:02.3130854Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9ddfc7292f2aef61.xml (deflated 37%) 2023-11-14T07:55:02.3132887Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a15e4efd5d263eac.xml (deflated 37%) 2023-11-14T07:55:02.3134793Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b654682239edce8f.xml (deflated 49%) 2023-11-14T07:55:02.3136629Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7d788a97a59a60cb.xml (deflated 49%) 2023-11-14T07:55:02.3139507Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a43625ed170e14eb.xml (deflated 48%) 2023-11-14T07:55:02.3141518Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e72bcd8eebbe7c4a.xml (deflated 50%) 2023-11-14T07:55:02.3143369Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8818dc53be66c9de.xml (deflated 38%) 2023-11-14T07:55:02.3145236Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-59b200171c6e4632.xml (deflated 38%) 2023-11-14T07:55:02.3147102Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d37151aa917be577.xml (deflated 38%) 2023-11-14T07:55:02.3149010Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8d5ac1e5db43336b.xml (deflated 45%) 2023-11-14T07:55:02.3150869Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f59e61224d84c4a5.xml (deflated 50%) 2023-11-14T07:55:02.3152721Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9875ec76c70a4b76.xml (deflated 38%) 2023-11-14T07:55:02.3154594Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a2b6faaae25d7eda.xml (deflated 38%) 2023-11-14T07:55:02.3156468Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-eed029f9ce0c3a50.xml (deflated 50%) 2023-11-14T07:55:02.3158318Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-669938266e207971.xml (deflated 38%) 2023-11-14T07:55:02.3160171Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6a779f7dd644cb8d.xml (deflated 37%) 2023-11-14T07:55:02.3162032Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2048e535312e21ab.xml (deflated 50%) 2023-11-14T07:55:02.3163888Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-43604f4117c0f378.xml (deflated 37%) 2023-11-14T07:55:02.3165724Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8209592a03710450.xml (deflated 38%) 2023-11-14T07:55:02.3167579Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ea76ab1237dd7443.xml (deflated 37%) 2023-11-14T07:55:02.3169439Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-18a87f42a73637bd.xml (deflated 37%) 2023-11-14T07:55:02.3171508Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-eebaa11b2ce77e26.xml (deflated 37%) 2023-11-14T07:55:02.3173376Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4de7e74389458b6f.xml (deflated 38%) 2023-11-14T07:55:02.3175246Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-473a739fea77af28.xml (deflated 38%) 2023-11-14T07:55:02.3177100Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9721dc5b6e63d7ac.xml (deflated 49%) 2023-11-14T07:55:02.3178964Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3de736ce44f5c765.xml (deflated 38%) 2023-11-14T07:55:02.3180809Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-26564f3f3c27afa1.xml (deflated 38%) 2023-11-14T07:55:02.3182826Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ea025a40ead71d78.xml (deflated 37%) 2023-11-14T07:55:02.3184779Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-18bc8a7bfdafb11f.xml (deflated 38%) 2023-11-14T07:55:02.3186771Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c424ccb251a10fc2.xml (deflated 36%) 2023-11-14T07:55:02.3188638Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fba8b472462bfab9.xml (deflated 36%) 2023-11-14T07:55:02.3190502Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7cd6733dbb63c8dd.xml (deflated 36%) 2023-11-14T07:55:02.3192361Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2383f9de29e28197.xml (deflated 37%) 2023-11-14T07:55:02.3194205Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0dc2e70d99b7efa7.xml (deflated 37%) 2023-11-14T07:55:02.3196080Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c10b81a8aeaf7df1.xml (deflated 37%) 2023-11-14T07:55:02.3197953Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d08c1ae8bd8fc51a.xml (deflated 37%) 2023-11-14T07:55:02.3199814Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-155c5659b6c8089c.xml (deflated 37%) 2023-11-14T07:55:02.3201648Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-17799ba153287501.xml (deflated 38%) 2023-11-14T07:55:02.3203500Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-555f76cf69e7e9b0.xml (deflated 37%) 2023-11-14T07:55:02.3205360Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d26d8440c50d7d4.xml (deflated 37%) 2023-11-14T07:55:02.3207221Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6fcb516cd9b02182.xml (deflated 37%) 2023-11-14T07:55:02.3209071Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-42b2c8d084c95ac3.xml (deflated 36%) 2023-11-14T07:55:02.3211118Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9629ddcf030ebc82.xml (deflated 36%) 2023-11-14T07:55:02.3212996Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-62956e2b2262f9ba.xml (deflated 36%) 2023-11-14T07:55:02.3214856Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6bb426223f451eed.xml (deflated 38%) 2023-11-14T07:55:02.3216700Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a18e581921bc7816.xml (deflated 37%) 2023-11-14T07:55:02.3218562Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7899e5a76a97bec3.xml (deflated 37%) 2023-11-14T07:55:02.3220431Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e92c065ac3c6b062.xml (deflated 49%) 2023-11-14T07:55:02.3222284Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e0b2a47db7e69abe.xml (deflated 49%) 2023-11-14T07:55:02.3224158Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a283eaa67dec5f73.xml (deflated 49%) 2023-11-14T07:55:02.3226021Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5d87833c473d128a.xml (deflated 37%) 2023-11-14T07:55:02.3227880Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0403948186d68eb6.xml (deflated 37%) 2023-11-14T07:55:02.3229837Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ab4e34e0058394f6.xml (deflated 37%) 2023-11-14T07:55:02.3231727Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c960229f352c7c44.xml (deflated 37%) 2023-11-14T07:55:02.3233590Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9d8b37aec26b9460.xml (deflated 37%) 2023-11-14T07:55:02.3235846Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1c3573ffc2ccf2af.xml (deflated 37%) 2023-11-14T07:55:02.3238187Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-308d420f23db9f99.xml (deflated 37%) 2023-11-14T07:55:02.3240547Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-42c19be00a5dde5d.xml (deflated 46%) 2023-11-14T07:55:02.3242913Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-aeb05881a689a0ae.xml (deflated 46%) 2023-11-14T07:55:02.3245270Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9c00359d2f2ad5d7.xml (deflated 46%) 2023-11-14T07:55:02.3247611Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-bd9c2ec6b11bd812.xml (deflated 47%) 2023-11-14T07:55:02.3249971Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-98be0c950ce3d608.xml (deflated 38%) 2023-11-14T07:55:02.3252835Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-6f174995aaacc804.xml (deflated 38%) 2023-11-14T07:55:02.3255195Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-36676aec30a0640b.xml (deflated 39%) 2023-11-14T07:55:02.3257558Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e9b1ade2f2c9a08e.xml (deflated 39%) 2023-11-14T07:55:02.3259899Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9ceb0b031b76fb3b.xml (deflated 39%) 2023-11-14T07:55:02.3262262Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-da551f06cd18cf9a.xml (deflated 39%) 2023-11-14T07:55:02.3264619Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2dad9d4b19d8c55f.xml (deflated 38%) 2023-11-14T07:55:02.3266975Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ce23b42ccc7a4f20.xml (deflated 38%) 2023-11-14T07:55:02.3269325Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d522aa5cda1e52a0.xml (deflated 39%) 2023-11-14T07:55:02.3271670Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-537485f08e5665a6.xml (deflated 39%) 2023-11-14T07:55:02.3274019Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-08e007eeb10e4584.xml (deflated 38%) 2023-11-14T07:55:02.3276366Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e27dcd7707287ed0.xml (deflated 38%) 2023-11-14T07:55:02.3278700Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d5e5379d5a735716.xml (deflated 38%) 2023-11-14T07:55:02.3281181Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2deb1e8904c1f4da.xml (deflated 39%) 2023-11-14T07:55:02.3283667Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-954ea1ef58339fd9.xml (deflated 38%) 2023-11-14T07:55:02.3286171Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1972a6c8ee4a2523.xml (deflated 38%) 2023-11-14T07:55:02.3288506Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d1478730b43bdd51.xml (deflated 38%) 2023-11-14T07:55:02.3291035Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3486281b05f36ce4.xml (deflated 38%) 2023-11-14T07:55:02.3293418Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3abfde64003fa5bc.xml (deflated 38%) 2023-11-14T07:55:02.3295778Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1d736a5bed77a1e0.xml (deflated 38%) 2023-11-14T07:55:02.3298119Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2d8ba98792ef78c4.xml (deflated 38%) 2023-11-14T07:55:02.3300475Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ac3c51ac7533d212.xml (deflated 38%) 2023-11-14T07:55:02.3302826Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-544f8b25fc95867b.xml (deflated 38%) 2023-11-14T07:55:02.3305180Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ffa0df891d195e51.xml (deflated 38%) 2023-11-14T07:55:02.3307535Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-056f1ff75d533499.xml (deflated 38%) 2023-11-14T07:55:02.3309869Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1b32183be0507117.xml (deflated 38%) 2023-11-14T07:55:02.3312217Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c8a5aa2e11480184.xml (deflated 38%) 2023-11-14T07:55:02.3314569Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0220fd9b76e939fb.xml (deflated 38%) 2023-11-14T07:55:02.3316919Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-23e538c7769b305d.xml (deflated 38%) 2023-11-14T07:55:02.3319258Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0b60cdd30256ff4b.xml (deflated 38%) 2023-11-14T07:55:02.3321605Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-275c77d0fc87fe45.xml (deflated 38%) 2023-11-14T07:55:02.3323964Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0e240d91cabe7bdc.xml (deflated 38%) 2023-11-14T07:55:02.3326325Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-fab2be0b9a9df88a.xml (deflated 38%) 2023-11-14T07:55:02.3328664Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8f5ccd4fd3265243.xml (deflated 39%) 2023-11-14T07:55:02.3331291Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4882a22d26acc6fc.xml (deflated 38%) 2023-11-14T07:55:02.3333690Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2d2c7a50fcff477e.xml (deflated 38%) 2023-11-14T07:55:02.3336169Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2a00f79a451a7e2f.xml (deflated 37%) 2023-11-14T07:55:02.3338503Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7a8f1f20b2b54ede.xml (deflated 38%) 2023-11-14T07:55:02.3340865Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dfeb4ead96a78248.xml (deflated 38%) 2023-11-14T07:55:02.3343229Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1bae3bd47bacf527.xml (deflated 38%) 2023-11-14T07:55:02.3345583Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5a32346e2eeda177.xml (deflated 38%) 2023-11-14T07:55:02.3347919Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-aae2cbf2c49c5dbe.xml (deflated 38%) 2023-11-14T07:55:02.3350294Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-092e0a9f7bfe0500.xml (deflated 38%) 2023-11-14T07:55:02.3352637Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2d2a7b744918e98a.xml (deflated 38%) 2023-11-14T07:55:02.3354979Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2a2400591e0a15ee.xml (deflated 38%) 2023-11-14T07:55:02.3357331Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3d4987868b40b8c5.xml (deflated 38%) 2023-11-14T07:55:02.3359660Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-27a43713ec062005.xml (deflated 38%) 2023-11-14T07:55:02.3362012Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a56f6de07b154c42.xml (deflated 38%) 2023-11-14T07:55:02.3364372Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-70b2e36b1bbb4d18.xml (deflated 39%) 2023-11-14T07:55:02.3366723Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-066d4f3231a3aa10.xml (deflated 48%) 2023-11-14T07:55:02.3369078Z adding: test/test-reports/python-pytest/distributed.fsdp.test_hsdp_dtensor_state_dict/distributed.fsdp.test_hsdp_dtensor_state_dict-8f0eea0d4e553999.xml (deflated 86%) 2023-11-14T07:55:02.3371570Z adding: test/test-reports/python-pytest/distributed.test_c10d_object_collectives/distributed.test_c10d_object_collectives-07fe253b82a84dd3.xml (deflated 77%) 2023-11-14T07:55:02.3373878Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_grad_acc/distributed.fsdp.test_fsdp_grad_acc-14fa9b049293084c.xml (deflated 76%) 2023-11-14T07:55:02.3376343Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_compile/distributed._composable.fully_shard.test_fully_shard_compile-643eb5b7cbc83e29.xml (deflated 40%) 2023-11-14T07:55:02.3378728Z adding: test/test-reports/python-pytest/distributed._tensor.test_tensor_ops/distributed._tensor.test_tensor_ops-3467c853b2c62add.xml (deflated 84%) 2023-11-14T07:55:02.3381086Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_flatten_params/distributed.fsdp.test_fsdp_flatten_params-66848bfa5b4fd36f.xml (deflated 79%) 2023-11-14T07:55:02.3383414Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_exec_order/distributed.fsdp.test_fsdp_exec_order-620ba67279a2ff2f.xml (deflated 79%) 2023-11-14T07:55:02.3385661Z adding: test/test-reports/python-pytest/distributed.fsdp.test_checkpoint_wrapper/distributed.fsdp.test_checkpoint_wrapper-be4e958195fd3827.xml (deflated 73%) 2023-11-14T07:55:02.3387907Z adding: test/test-reports/python-pytest/distributed.test_data_parallel/distributed.test_data_parallel-f2f2d9298924a9e6.xml (deflated 87%) 2023-11-14T07:55:02.3390075Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_mixed_precision/distributed.fsdp.test_fsdp_mixed_precision-2c3f4640744c2317.xml (deflated 92%) 2023-11-14T07:55:02.3392466Z adding: test/test-reports/python-pytest/distributed.optim.test_zero_redundancy_optimizer/distributed.optim.test_zero_redundancy_optimizer-9e6c4126c368043f.xml (deflated 91%) 2023-11-14T07:55:02.3394808Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-268b3b5934107f09.xml (deflated 89%) 2023-11-14T07:55:02.3396773Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f1467e19319f3abf.xml (deflated 50%) 2023-11-14T07:55:02.3398556Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f99873a8b64b4112.xml (deflated 37%) 2023-11-14T07:55:02.3400326Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e264d0bf565831e2.xml (deflated 37%) 2023-11-14T07:55:02.3402102Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-91ad3e01dfe01af9.xml (deflated 50%) 2023-11-14T07:55:02.3403864Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-751976b18dc0b3fb.xml (deflated 50%) 2023-11-14T07:55:02.3405633Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ecdeee07b193347e.xml (deflated 38%) 2023-11-14T07:55:02.3407424Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ba0d15f84d9f1b79.xml (deflated 38%) 2023-11-14T07:55:02.3409201Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f3b20d42ce3cf55a.xml (deflated 37%) 2023-11-14T07:55:02.3411133Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-64443e117acf039e.xml (deflated 38%) 2023-11-14T07:55:02.3412922Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-8b3f82e8fcbacb2f.xml (deflated 38%) 2023-11-14T07:55:02.3414699Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7e039e5998f1aa72.xml (deflated 37%) 2023-11-14T07:55:02.3416470Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6a6799aa080e230c.xml (deflated 38%) 2023-11-14T07:55:02.3418227Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-be7d05309da10dc2.xml (deflated 38%) 2023-11-14T07:55:02.3420002Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4e0f1b52fb8d4e03.xml (deflated 38%) 2023-11-14T07:55:02.3421781Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4d9cdab55f00e0ff.xml (deflated 37%) 2023-11-14T07:55:02.3423546Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-35f277fe707ad56f.xml (deflated 50%) 2023-11-14T07:55:02.3425318Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-40ecae7bce1109c4.xml (deflated 37%) 2023-11-14T07:55:02.3427089Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7b4427d11ef62518.xml (deflated 49%) 2023-11-14T07:55:02.3428861Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4acf4b3cb1d2587c.xml (deflated 49%) 2023-11-14T07:55:02.3430748Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-10d584d7fbfed3b6.xml (deflated 37%) 2023-11-14T07:55:02.3432544Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c1870bb3a5ba3e7f.xml (deflated 37%) 2023-11-14T07:55:02.3434455Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-67dc1d7367dc4848.xml (deflated 37%) 2023-11-14T07:55:02.3436219Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-67713b8dacd78751.xml (deflated 37%) 2023-11-14T07:55:02.3437972Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-94b1bf5af6ea24e5.xml (deflated 37%) 2023-11-14T07:55:02.3439742Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-dd8a0186f584ad5f.xml (deflated 37%) 2023-11-14T07:55:02.3441517Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-70d667a49d69af8f.xml (deflated 37%) 2023-11-14T07:55:02.3443284Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6413d463f2948881.xml (deflated 37%) 2023-11-14T07:55:02.3445039Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fb5dd5ed3a719c47.xml (deflated 37%) 2023-11-14T07:55:02.3446809Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-613f79db6426f427.xml (deflated 37%) 2023-11-14T07:55:02.3448576Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-646d2ae25568c4e6.xml (deflated 37%) 2023-11-14T07:55:02.3450321Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-08d2b8634fd88d6a.xml (deflated 37%) 2023-11-14T07:55:02.3452268Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-42f550a73d358161.xml (deflated 37%) 2023-11-14T07:55:02.3454045Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6e7c70194b41c0b9.xml (deflated 37%) 2023-11-14T07:55:02.3455813Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5876c00b63735954.xml (deflated 38%) 2023-11-14T07:55:02.3457564Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-93aafd32384f93f9.xml (deflated 37%) 2023-11-14T07:55:02.3459334Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a8788964a26cd7a6.xml (deflated 37%) 2023-11-14T07:55:02.3461099Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-123a2df2b2a7afe3.xml (deflated 37%) 2023-11-14T07:55:02.3462873Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1c5946f1ca39c7e0.xml (deflated 37%) 2023-11-14T07:55:02.3464626Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6ae08deeb6b2bb12.xml (deflated 36%) 2023-11-14T07:55:02.3466403Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5d1f1a5c1d0f749d.xml (deflated 36%) 2023-11-14T07:55:02.3468172Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5d6561ee9a55fa80.xml (deflated 37%) 2023-11-14T07:55:02.3469930Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3f0b19afb43f3742.xml (deflated 36%) 2023-11-14T07:55:02.3471696Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-53a9e505e2a67cb4.xml (deflated 35%) 2023-11-14T07:55:02.3473461Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-91fe402a52b5fdca.xml (deflated 37%) 2023-11-14T07:55:02.3475230Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-11907118ca017fd2.xml (deflated 37%) 2023-11-14T07:55:02.3477113Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a8974b9a03f75bd7.xml (deflated 37%) 2023-11-14T07:55:02.3478910Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-8796eb56924fc1fb.xml (deflated 37%) 2023-11-14T07:55:02.3480671Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d22495422ca5581c.xml (deflated 37%) 2023-11-14T07:55:02.3482570Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-adde2123d6c4371f.xml (deflated 37%) 2023-11-14T07:55:02.3484373Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fe78b072edd0cb0b.xml (deflated 37%) 2023-11-14T07:55:02.3486136Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-67606a46f28fdd54.xml (deflated 37%) 2023-11-14T07:55:02.3487904Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e8ca8ce84cd7892b.xml (deflated 37%) 2023-11-14T07:55:02.3489681Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f52dcc3303d028b1.xml (deflated 37%) 2023-11-14T07:55:02.3491616Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6e6f7e28c5b790d0.xml (deflated 37%) 2023-11-14T07:55:02.3493389Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f5ce0430c4fd4ab0.xml (deflated 37%) 2023-11-14T07:55:02.3495175Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7bfdfdfd5557da51.xml (deflated 38%) 2023-11-14T07:55:02.3496935Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-268ad8358ba3ff8c.xml (deflated 37%) 2023-11-14T07:55:02.3498704Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5a75818c11819c2e.xml (deflated 37%) 2023-11-14T07:55:02.3500461Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-333ef343e5481260.xml (deflated 37%) 2023-11-14T07:55:02.3502232Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-52b3f8ce94e030ac.xml (deflated 38%) 2023-11-14T07:55:02.3503976Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-03284158e54eff7f.xml (deflated 37%) 2023-11-14T07:55:02.3505756Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2e67e52ec4baedd6.xml (deflated 37%) 2023-11-14T07:55:02.3507531Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f5554f45ab28f516.xml (deflated 37%) 2023-11-14T07:55:02.3509303Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-bf9f2bfe761fcce2.xml (deflated 37%) 2023-11-14T07:55:02.3511066Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e732b68cfcb5ac78.xml (deflated 38%) 2023-11-14T07:55:02.3512844Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-cd1a15aa2f37c188.xml (deflated 38%) 2023-11-14T07:55:02.3514620Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b582e62f2e954b9b.xml (deflated 37%) 2023-11-14T07:55:02.3516398Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1adfcc26d2d08cd2.xml (deflated 37%) 2023-11-14T07:55:02.3518158Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e5d553199a1500cd.xml (deflated 37%) 2023-11-14T07:55:02.3519914Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2991511014d52f72.xml (deflated 37%) 2023-11-14T07:55:02.3521675Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-142bd3a9672c4924.xml (deflated 37%) 2023-11-14T07:55:02.3523424Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-79a0df6bc7f41c01.xml (deflated 37%) 2023-11-14T07:55:02.3525326Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1bfd48ac77f00a4a.xml (deflated 37%) 2023-11-14T07:55:02.3526226Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0483ee92bfc6d022.xml (deflated 37%) 2023-11-14T07:55:02.3527068Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-00084cc27b366b11.xml (deflated 37%) 2023-11-14T07:55:02.3528061Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2d24c52dee9e9723.xml (deflated 37%) 2023-11-14T07:55:02.3528922Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7a8f00c1e15fea81.xml (deflated 37%) 2023-11-14T07:55:02.3529775Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ede1dc2bb9e2f7cc.xml (deflated 59%) 2023-11-14T07:55:02.3530637Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ae7254175e25a70f.xml (deflated 59%) 2023-11-14T07:55:02.3531779Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-dfcaa8157f89d4c4.xml (deflated 59%) 2023-11-14T07:55:02.3532625Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e256f6dc495414a8.xml (deflated 59%) 2023-11-14T07:55:02.3533492Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5b4111c2bfd6cf3d.xml (deflated 38%) 2023-11-14T07:55:02.3534334Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-047c3ae60583352e.xml (deflated 36%) 2023-11-14T07:55:02.3535197Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fd7d0b523085dadf.xml (deflated 36%) 2023-11-14T07:55:02.3536295Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_hybrid_shard/distributed.fsdp.test_fsdp_hybrid_shard-072a76b25c1e7b0e.xml (deflated 75%) 2023-11-14T07:55:02.3537289Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-0817b93f3b92a31f.xml (deflated 80%) 2023-11-14T07:55:02.3538351Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_overlap/distributed.fsdp.test_fsdp_overlap-9a0a01ee6a3f7524.xml (deflated 51%) 2023-11-14T07:55:02.3539471Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_checkpoint/distributed.checkpoint.test_checkpoint-a8e46aea9531e803.xml (deflated 79%) 2023-11-14T07:55:02.3540515Z adding: test/test-reports/python-pytest/distributed._tensor.test_matrix_ops/distributed._tensor.test_matrix_ops-6a3caa6885c93f7e.xml (deflated 73%) 2023-11-14T07:55:02.3541456Z adding: test/test-reports/python-pytest/distributed._tensor.test_init/distributed._tensor.test_init-5738bdc1b19115ad.xml (deflated 81%) 2023-11-14T07:55:02.3542535Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-737c8265f2e8662c.xml (deflated 87%) 2023-11-14T07:55:02.3543783Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_resharding/distributed.checkpoint.test_dtensor_resharding-b5b6bf1d79b4bad8.xml (deflated 69%) 2023-11-14T07:55:02.3544996Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-a8b300dc5e3fec05.xml (deflated 72%) 2023-11-14T07:55:02.3546157Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-ddeb11fcac328817.xml (deflated 79%) 2023-11-14T07:55:02.3547129Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-9819138c1f8f1631.xml (deflated 48%) 2023-11-14T07:55:02.3548102Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-29861a67cfd990ab.xml (deflated 36%) 2023-11-14T07:55:02.3549130Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-2b6c4338c2eec26c.xml (deflated 35%) 2023-11-14T07:55:02.3550207Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-c15041d4593d5466.xml (deflated 35%) 2023-11-14T07:55:02.3551236Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-f60cea896c000abc.xml (deflated 35%) 2023-11-14T07:55:02.3552334Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-b3099a6017d20d02.xml (deflated 35%) 2023-11-14T07:55:02.3553385Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_apply/distributed.fsdp.test_fsdp_apply-1b7a62395dbfc4ee.xml (deflated 56%) 2023-11-14T07:55:02.3554605Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_state_dict_utils/distributed.checkpoint.test_state_dict_utils-83175acbffac1282.xml (deflated 73%) 2023-11-14T07:55:02.3555830Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_optim_state/distributed.checkpoint.test_fsdp_optim_state-cd9ab0641ddd460c.xml (deflated 53%) 2023-11-14T07:55:02.3557010Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_model_state/distributed.checkpoint.test_fsdp_model_state-b3375ba84e1920ea.xml (deflated 55%) 2023-11-14T07:55:02.3558224Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_init/distributed._shard.sharded_tensor.ops.test_init-865c1a5863349c95.xml (deflated 73%) 2023-11-14T07:55:02.3559451Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_example/distributed.elastic.timer.local_timer_example-0af2515e80879cce.xml (deflated 49%) 2023-11-14T07:55:02.3560833Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/distributed._shard.sharded_tensor.test_sharded_tensor_reshard-38f54147bc8f7646.xml (deflated 68%) 2023-11-14T07:55:02.3562218Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion-657691f082c04fe1.xml (deflated 40%) 2023-11-14T07:55:02.3563442Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_checkpoint/distributed.checkpoint.test_dtensor_checkpoint-9f1f3234516a7a56.xml (deflated 41%) 2023-11-14T07:55:02.3564539Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-fe6d9172d8a8021e.xml (deflated 39%) 2023-11-14T07:55:02.3565705Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_wrapping/distributed.fsdp.test_fsdp_multiple_wrapping-a07db3529b05466e.xml (deflated 40%) 2023-11-14T07:55:02.3566871Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_test/distributed.elastic.timer.local_timer_test-8f1271f162005636.xml (deflated 80%) 2023-11-14T07:55:02.3567914Z adding: test/test-reports/python-pytest/distributed.rpc.test_share_memory/distributed.rpc.test_share_memory-1de471d734f146c6.xml (deflated 48%) 2023-11-14T07:55:02.3569024Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_stream/distributed.pipeline.sync.test_stream-7357dc4da0d6a5a6.xml (deflated 86%) 2023-11-14T07:55:02.3570121Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_copy/distributed.pipeline.sync.test_copy-bc6f0cc1246dfd99.xml (deflated 69%) 2023-11-14T07:55:02.3571464Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_leak/distributed.pipeline.sync.skip.test_leak-493f4061f1c3091b.xml (deflated 75%) 2023-11-14T07:55:02.3572532Z adding: test/test-reports/python-pytest/distributed._tensor.test_op_strategy/distributed._tensor.test_op_strategy-48b17fd583ee9961.xml (deflated 81%) 2023-11-14T07:55:02.3573607Z adding: test/test-reports/python-pytest/distributed.elastic.events.lib_test/distributed.elastic.events.lib_test-5697ad14e7b88117.xml (deflated 76%) 2023-11-14T07:55:02.3574799Z adding: test/test-reports/python-pytest/distributed.nn.jit.test_instantiator/distributed.nn.jit.test_instantiator-78c766a4eda672bf.xml (deflated 61%) 2023-11-14T07:55:02.3576016Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-f123c40e7c2456a9.xml (deflated 37%) 2023-11-14T07:55:02.3577233Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_traverse/distributed.checkpoint.test_traverse-791ae77332741f5e.xml (deflated 75%) 2023-11-14T07:55:02.3578348Z adding: test/test-reports/python-pytest/distributed._composable.test_checkpoint/distributed._composable.test_checkpoint-92f0c90db6b8e3b5.xml (deflated 69%) 2023-11-14T07:55:02.3579546Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_portal/distributed.pipeline.sync.skip.test_portal-73a01f4cf7c721e4.xml (deflated 80%) 2023-11-14T07:55:02.3580640Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_phony/distributed.pipeline.sync.test_phony-78aa7884d3ca089e.xml (deflated 64%) 2023-11-14T07:55:02.3612992Z ##[group]Run # Remove any previous test reports if they exist 2023-11-14T07:55:02.3613314Z # Remove any previous test reports if they exist 2023-11-14T07:55:02.3613488Z rm -f logs-*.zip 2023-11-14T07:55:02.3614041Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2023-11-14T07:55:02.3614292Z # so check to see if the file exists first 2023-11-14T07:55:02.3614500Z if [ -f 'usage_log.txt' ]; then 2023-11-14T07:55:02.3614780Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2023-11-14T07:55:02.3614910Z fi 2023-11-14T07:55:02.3615166Z if ls test/**/*.log 1> /dev/null 2>&1; then 2023-11-14T07:55:02.3615449Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2023-11-14T07:55:02.3615600Z fi 2023-11-14T07:55:02.3623874Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:55:02.3624023Z env: 2023-11-14T07:55:02.3624195Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:02.3624477Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:02.3624963Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:02.3625375Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019 2023-11-14T07:55:02.3625524Z ##[endgroup] 2023-11-14T07:55:02.5253615Z adding: usage_log.txt (deflated 96%) 2023-11-14T07:55:02.5390432Z adding: test/test-reports/distributed.rpc.test_share_memory_1.1_1b953cb39c46370b_.log (deflated 56%) 2023-11-14T07:55:02.5391952Z adding: test/test-reports/distributed.test_distributed_spawn_1.2_6954fa4c4f370602_.log (deflated 13%) 2023-11-14T07:55:02.5393372Z adding: test/test-reports/distributed._spmd.test_graph_utils_1.1_448000892674d5c9_.log (stored 0%) 2023-11-14T07:55:02.5394792Z adding: test/test-reports/distributed.test_distributed_spawn_1.2_d243e10469c8e9c2_.log (deflated 12%) 2023-11-14T07:55:02.5396211Z adding: test/test-reports/distributed.test_collective_utils_1.1_2bb22d83665ed083_.log (stored 0%) 2023-11-14T07:55:02.5412934Z adding: test/test-reports/distributed.test_distributed_spawn_1.2_a419aa61a4bf8873_.log (deflated 95%) 2023-11-14T07:55:02.5433821Z adding: test/test-reports/distributed.test_distributed_spawn_1.2_02b673c50f1d3d59_.log (deflated 95%) 2023-11-14T07:55:02.5457536Z adding: test/test-reports/distributed.test_distributed_spawn_1.2_ce38aa1f69712c59_.log (deflated 95%) 2023-11-14T07:55:02.5480743Z adding: test/test-reports/distributed.test_distributed_spawn_1.2_49769ddc22e23e17_.log (deflated 95%) 2023-11-14T07:55:02.5489871Z adding: test/test-reports/distributed.test_store_1.1_1dec0fb1c10742a9_.log (deflated 95%) 2023-11-14T07:55:02.5556459Z adding: test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_8222374d7fd53e54_.log (deflated 92%) 2023-11-14T07:55:02.5558356Z adding: test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_b53da69f833cc4c6_.log (deflated 91%) 2023-11-14T07:55:02.5560354Z adding: test/test-reports/distributed.rpc.test_tensorpipe_agent_1.1_ea1ae83cfd92a020_.log (stored 0%) 2023-11-14T07:55:02.5561875Z adding: test/test-reports/distributed.test_multi_threaded_pg_1.1_635ac9df2e2dbdde_.log (deflated 85%) 2023-11-14T07:55:02.5580607Z adding: test/test-reports/distributed.test_c10d_nccl_1.1_3be3173cab9a1fc5_.log (deflated 95%) 2023-11-14T07:55:02.5582307Z adding: test/test-reports/distributed.checkpoint.test_checkpoint_1.1_23212d6d52d68fff_.log (deflated 80%) 2023-11-14T07:55:02.5589273Z adding: test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_2.2_bb2240a166d6047a_.log (deflated 96%) 2023-11-14T07:55:02.5590906Z adding: test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_5d62ceeca88f7d07_.log (deflated 78%) 2023-11-14T07:55:02.5592438Z adding: test/test-reports/distributed._tensor.test_matrix_ops_1.1_1e5fa2fe17809d44_.log (deflated 72%) 2023-11-14T07:55:02.5593941Z adding: test/test-reports/distributed.test_c10d_object_collectives_1.1_1d83110884a90e03_.log (deflated 74%) 2023-11-14T07:55:02.5595670Z adding: test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_cb25fe4204c1ffbd_.log (deflated 96%) 2023-11-14T07:55:02.5601048Z adding: test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_9ddfd62101668295_.log (deflated 94%) 2023-11-14T07:55:02.5603833Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_compile_1.1_928f8a9a29d1b5a7_.log (deflated 94%) 2023-11-14T07:55:02.5606047Z adding: test/test-reports/distributed._tensor.test_tensor_ops_1.1_f9624234f34a6c61_.log (deflated 79%) 2023-11-14T07:55:02.5608030Z adding: test/test-reports/distributed._tensor.test_init_1.1_7b115b83f4a0cec4_.log (deflated 69%) 2023-11-14T07:55:02.5610146Z adding: test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_f60b7b2b4b418f45_.log (deflated 83%) 2023-11-14T07:55:02.5612428Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_72717c95532b386d_.log (deflated 82%) 2023-11-14T07:55:02.5614286Z adding: test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_cd6c8a3c178e18a3_.log (deflated 80%) 2023-11-14T07:55:02.5615829Z adding: test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_cb765f93f2a202cb_.log (deflated 73%) 2023-11-14T07:55:02.5617407Z adding: test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_2e1d514b518b636b_.log (deflated 71%) 2023-11-14T07:55:02.5618861Z adding: test/test-reports/distributed.test_data_parallel_1.1_82ef56b1e5dfddb7_.log (deflated 87%) 2023-11-14T07:55:02.5643117Z adding: test/test-reports/distributed._spmd.test_tracing_1.1_0bb221544268a705_.log (stored 0%) 2023-11-14T07:55:02.5645105Z adding: test/test-reports/distributed.test_c10d_spawn_ucc_1.1_f38cf3c1188558dc_.log (deflated 89%) 2023-11-14T07:55:02.5647485Z adding: test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_abc742475dcd9b44_.log (deflated 97%) 2023-11-14T07:55:02.5649583Z adding: test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_b66088cfc7293888_.log (deflated 78%) 2023-11-14T07:55:02.5651628Z adding: test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_9c3167609316da8e_.log (deflated 88%) 2023-11-14T07:55:02.5653558Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_b0ad30105458d6dc_.log (deflated 86%) 2023-11-14T07:55:02.5655054Z adding: test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_8f67f640c7fd6813_.log (deflated 61%) 2023-11-14T07:55:02.5656609Z adding: test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_3b4526746de6eac5_.log (deflated 68%) 2023-11-14T07:55:02.5658464Z adding: test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_c171de1fae27011c_.log (deflated 73%) 2023-11-14T07:55:02.5661410Z adding: test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_5ab508dd65919998_.log (deflated 66%) 2023-11-14T07:55:02.5663039Z adding: test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_3a8de01c3537b5ac_.log (deflated 77%) 2023-11-14T07:55:02.5664841Z adding: test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_0a94a8c2fd39d352_.log (deflated 59%) 2023-11-14T07:55:02.5666490Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_f769f941e7e30c94_.log (deflated 66%) 2023-11-14T07:55:02.5668290Z adding: test/test-reports/distributed.elastic.timer.local_timer_example_1.1_1588dbbc2ee073d6_.log (deflated 59%) 2023-11-14T07:55:02.5670014Z adding: test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_reshard_1.1_f0f159d2b6036333_.log (deflated 63%) 2023-11-14T07:55:02.5671789Z adding: test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_c79d50a680af6e2a_.log (deflated 55%) 2023-11-14T07:55:02.5673508Z adding: test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_476c72372bdc71cd_.log (deflated 54%) 2023-11-14T07:55:02.5675065Z adding: test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_8057e0e65c3589cb_.log (deflated 54%) 2023-11-14T07:55:02.5676626Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_dbf4284801bb1e2b_.log (deflated 56%) 2023-11-14T07:55:02.5678216Z adding: test/test-reports/distributed.elastic.timer.local_timer_test_1.1_22941d1297b02b8a_.log (deflated 77%) 2023-11-14T07:55:02.5679784Z adding: test/test-reports/distributed.pipeline.sync.test_stream_1.1_64525298bc454ca2_.log (deflated 81%) 2023-11-14T07:55:02.5681302Z adding: test/test-reports/distributed.pipeline.sync.test_copy_1.1_314456196a3318fc_.log (deflated 64%) 2023-11-14T07:55:02.5682920Z adding: test/test-reports/distributed.pipeline.sync.skip.test_leak_1.1_9d12d5f6fecf9550_.log (deflated 72%) 2023-11-14T07:55:02.5684445Z adding: test/test-reports/distributed._tensor.test_op_strategy_1.1_0ea0f8fd21c07057_.log (deflated 77%) 2023-11-14T07:55:02.5685936Z adding: test/test-reports/distributed.elastic.events.lib_test_1.1_cb7840f0821b4016_.log (deflated 72%) 2023-11-14T07:55:02.5687444Z adding: test/test-reports/distributed.nn.jit.test_instantiator_1.1_f9a32daeb029829c_.log (deflated 63%) 2023-11-14T07:55:02.5689005Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_2f3711ec3f23f16b_.log (deflated 55%) 2023-11-14T07:55:02.5690567Z adding: test/test-reports/distributed.checkpoint.test_traverse_1.1_a8c3aa361caa42e3_.log (deflated 71%) 2023-11-14T07:55:02.5692359Z adding: test/test-reports/distributed._composable.test_checkpoint_1.1_029f98d9627b9fb2_.log (deflated 67%) 2023-11-14T07:55:02.5693949Z adding: test/test-reports/distributed.pipeline.sync.skip.test_portal_1.1_00960e18abe2ded1_.log (deflated 75%) 2023-11-14T07:55:02.5695517Z adding: test/test-reports/distributed.pipeline.sync.test_phony_1.1_e37b13b21d9d4804_.log (deflated 62%) 2023-11-14T07:55:02.5696982Z adding: test/test-reports/distributed._tensor.test_optimizers_1.1_946b5e06fb690ac7_.log (stored 0%) 2023-11-14T07:55:02.5698522Z adding: test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_751d3fc6290fe2ea_.log (stored 0%) 2023-11-14T07:55:02.5772960Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-11-14T07:55:02.5773515Z with: 2023-11-14T07:55:02.5773943Z s3-prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:55:02.5774513Z retention-days: 14 2023-11-14T07:55:02.5774928Z if-no-files-found: warn 2023-11-14T07:55:02.5775379Z path: test-jsons-*.zip 2023-11-14T07:55:02.5775783Z name: artifact 2023-11-14T07:55:02.5776167Z s3-bucket: gha-artifacts 2023-11-14T07:55:02.5776601Z region: us-east-1 2023-11-14T07:55:02.5776949Z env: 2023-11-14T07:55:02.5777284Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:02.5777845Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:02.5778759Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:02.5779574Z ##[endgroup] 2023-11-14T07:55:03.0260108Z NOTE: s3-prefix specified, ignoring name parameter 2023-11-14T07:55:03.0260879Z With the provided path, there will be 1 file uploaded 2023-11-14T07:55:03.0261876Z Uploading to s3 prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:55:03.0273070Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019.zip 2023-11-14T07:55:03.1860348Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019.zip 2023-11-14T07:55:03.2033415Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-11-14T07:55:03.2033952Z with: 2023-11-14T07:55:03.2034381Z s3-prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:55:03.2034967Z retention-days: 14 2023-11-14T07:55:03.2035365Z if-no-files-found: error 2023-11-14T07:55:03.2035817Z path: test-reports-*.zip 2023-11-14T07:55:03.2036248Z name: artifact 2023-11-14T07:55:03.2036609Z s3-bucket: gha-artifacts 2023-11-14T07:55:03.2037041Z region: us-east-1 2023-11-14T07:55:03.2037409Z env: 2023-11-14T07:55:03.2037725Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:03.2038279Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:03.2039220Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:03.2040014Z ##[endgroup] 2023-11-14T07:55:03.6480191Z NOTE: s3-prefix specified, ignoring name parameter 2023-11-14T07:55:03.6480999Z With the provided path, there will be 1 file uploaded 2023-11-14T07:55:03.6481804Z Uploading to s3 prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:55:03.6493135Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019.zip 2023-11-14T07:55:03.9565240Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019.zip 2023-11-14T07:55:03.9759893Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-11-14T07:55:03.9760483Z with: 2023-11-14T07:55:03.9760911Z s3-prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:55:03.9761533Z retention-days: 14 2023-11-14T07:55:03.9761979Z if-no-files-found: ignore 2023-11-14T07:55:03.9762422Z path: logs-*.zip 2023-11-14T07:55:03.9762852Z name: artifact 2023-11-14T07:55:03.9763272Z s3-bucket: gha-artifacts 2023-11-14T07:55:03.9763709Z region: us-east-1 2023-11-14T07:55:03.9764107Z env: 2023-11-14T07:55:03.9764466Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:03.9765030Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:03.9766011Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:03.9766851Z ##[endgroup] 2023-11-14T07:55:04.4219804Z NOTE: s3-prefix specified, ignoring name parameter 2023-11-14T07:55:04.4220776Z With the provided path, there will be 1 file uploaded 2023-11-14T07:55:04.4221555Z Uploading to s3 prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:55:04.4232054Z Starting upload of logs-test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019.zip 2023-11-14T07:55:04.6567126Z Finished upload of logs-test-distributed-1-3-linux.8xlarge.nvidia.gpu_18651430019.zip 2023-11-14T07:55:04.6734738Z ##[group]Run # shellcheck disable=SC2156 2023-11-14T07:55:04.6735350Z # shellcheck disable=SC2156 2023-11-14T07:55:04.6736356Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2023-11-14T07:55:04.6745181Z shell: /usr/bin/bash -e {0} 2023-11-14T07:55:04.6745626Z env: 2023-11-14T07:55:04.6745965Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:04.6746535Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:04.6747470Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:04.6748274Z ##[endgroup] 2023-11-14T07:55:04.9964806Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-11-14T07:55:04.9965593Z with: 2023-11-14T07:55:04.9966128Z name: coredumps-distributed-1-3-linux.8xlarge.nvidia.gpu 2023-11-14T07:55:04.9966789Z retention-days: 14 2023-11-14T07:55:04.9967206Z if-no-files-found: ignore 2023-11-14T07:55:04.9967652Z path: ./**/core.[1-9]* 2023-11-14T07:55:04.9968059Z s3-bucket: gha-artifacts 2023-11-14T07:55:04.9968493Z region: us-east-1 2023-11-14T07:55:04.9968858Z env: 2023-11-14T07:55:04.9969169Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:04.9969722Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:04.9970648Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:04.9971987Z ##[endgroup] 2023-11-14T07:55:16.2060254Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2023-11-14T07:55:16.2324712Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2023-11-14T07:55:16.2325423Z with: 2023-11-14T07:55:16.2325736Z env: 2023-11-14T07:55:16.2326073Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:16.2326619Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:16.2327543Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:16.2328352Z ##[endgroup] 2023-11-14T07:55:16.2354934Z ##[group]Run set -eou pipefail 2023-11-14T07:55:16.2355448Z set -eou pipefail 2023-11-14T07:55:16.2355880Z  2023-11-14T07:55:16.2356510Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2023-11-14T07:55:16.2357297Z for _ in $(seq 1440); do 2023-11-14T07:55:16.2357895Z  # Break if no ssh session exists anymore 2023-11-14T07:55:16.2358516Z  if [ "$(who)" = "" ]; then 2023-11-14T07:55:16.2358999Z  break 2023-11-14T07:55:16.2359434Z  fi 2023-11-14T07:55:16.2359786Z  echo "." 2023-11-14T07:55:16.2360186Z  sleep 5 2023-11-14T07:55:16.2360569Z done 2023-11-14T07:55:16.2368898Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:55:16.2369501Z env: 2023-11-14T07:55:16.2369841Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:16.2370403Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:16.2371609Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:16.2372426Z ##[endgroup] 2023-11-14T07:55:16.2399852Z Holding runner for 2 hours until all ssh sessions have logged out 2023-11-14T07:55:16.2458840Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2023-11-14T07:55:16.2459823Z # ignore expansion of "docker ps -q" since it could be empty 2023-11-14T07:55:16.2460571Z # shellcheck disable=SC2046 2023-11-14T07:55:16.2461159Z docker stop $(docker ps -q) || true 2023-11-14T07:55:16.2461743Z # Prune all of the docker images 2023-11-14T07:55:16.2462302Z docker system prune -af 2023-11-14T07:55:16.2470387Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:55:16.2471010Z env: 2023-11-14T07:55:16.2471336Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:16.2471905Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:16.2472841Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:16.2473641Z ##[endgroup] 2023-11-14T07:55:16.6391588Z 99f90b6e6ff3 2023-11-14T07:55:17.1643641Z Deleted Containers: 2023-11-14T07:55:17.1644314Z 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:17.1644886Z 2023-11-14T07:55:23.4113493Z Deleted Images: 2023-11-14T07:55:23.4115160Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T07:55:23.4119138Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9@sha256:155cc70f81c46492d7369e444f8774cfbc3ff3ec6c7d0217deb03d9c42253389 2023-11-14T07:55:23.4121018Z deleted: sha256:2e9cd9f64120afd8fb911e301ea6adc44cad71e8bfde2785095ab22e8b7f1f32 2023-11-14T07:55:23.4122415Z deleted: sha256:512838e73d4f3d028fbbe7052c9d1f8206ef2659c60359ab52b34e4cc35c3992 2023-11-14T07:55:23.4123950Z deleted: sha256:a9d8f9d40a750124a6f8093111d44b5278d65dd1a2f791289f48f8724dbb459e 2023-11-14T07:55:23.4125021Z deleted: sha256:7be8ab7c9adc25b2f1ba0f4c4ca11050fd97f8678c8bbc4074e9532c54e0d7f9 2023-11-14T07:55:23.4126079Z deleted: sha256:83a338be1d118af6b97d3695eb7f03a4a6f3b727fd1515e133df986281bc45aa 2023-11-14T07:55:23.4127164Z deleted: sha256:853b0173ff3fe3deb14968f3ea4a3159098a702b026f1813e4777ab0c1bd1b15 2023-11-14T07:55:23.4128748Z deleted: sha256:146b0656459f08e735bf0326b9c46dcb5add40d800a3f0fa7f753d73fba2bc48 2023-11-14T07:55:23.4130460Z deleted: sha256:c5c9d92208787809f5e8e4e2c199d194a1c6d9cc1e625b1a4c4681be00e25f08 2023-11-14T07:55:23.4132103Z deleted: sha256:506b175b96c4889f411b3d98ba1bcc3a695d189d9f75859b7c53d1d7c6c84c62 2023-11-14T07:55:23.4133178Z deleted: sha256:bd30c7b38cc8f59afc412d106e97b45cfdc9f076e11a734c19b045153eea3dbc 2023-11-14T07:55:23.4134253Z deleted: sha256:f86fe52700a5bdaadba6e76e1d72209470b879c254d5c2f80a931b7de263a640 2023-11-14T07:55:23.4135314Z deleted: sha256:128374ff7ddd2bb50e4dd80e5a8029dc3c6498f66efed461a5db0a38eb830df4 2023-11-14T07:55:23.4136397Z deleted: sha256:4140552c990c771f35f0bfffa628800ad4caffebd171f152bd3f6909a33afefa 2023-11-14T07:55:23.4137470Z deleted: sha256:f0484d0bf7d841b21262a867fd30ff4a6cc1fd974036859e4100ec576c9117d2 2023-11-14T07:55:23.4138568Z deleted: sha256:fb583a2e1b7d9574e950b005fdc068b75673b4a719b41972da5f09c4287903f7 2023-11-14T07:55:23.4139632Z deleted: sha256:a5d7b2a2a66e710c2b300ffd3208b143f26caaaebbe936f70d48d748d888d970 2023-11-14T07:55:23.4140719Z deleted: sha256:814a71ffc7f1e74711aded1614eafb2e2d475994244b16bcf5aee091e8505aa7 2023-11-14T07:55:23.4141801Z deleted: sha256:a595f3bb7d57f50c1c0023fe87259e1fb2f04073eafbe45a4f41df19a137c840 2023-11-14T07:55:23.4142821Z deleted: sha256:f00286388f7c580b213816f52863390af7e56967b92b4e4c8421167eda1e2fd9 2023-11-14T07:55:23.4143841Z deleted: sha256:79652a8cad146a70172780b29b4d868800098f6dfa17da14f851143254a9786e 2023-11-14T07:55:23.4144889Z deleted: sha256:dbb5eb27c0b50d5693a5a24d3b114fa312270d8d08ea0bd7e89f346437979a50 2023-11-14T07:55:23.4145943Z deleted: sha256:1d75aa6c5012f2747eb72b6861274c2f46f89675170aa4c7000b295a487b7bfd 2023-11-14T07:55:23.4146964Z deleted: sha256:c85b89d91368af1bc72c26119c221e7daa8a195b18fde34a88b7c9037208c401 2023-11-14T07:55:23.4148024Z deleted: sha256:04e30a2271f62fb8c83dabac48f382c43155c877ba30fae77dbe53acb902eb82 2023-11-14T07:55:23.4149102Z deleted: sha256:559cad94e52b67a628fb575fe5d0a1835cccc9d5a9f65d73a201003c6e7092cf 2023-11-14T07:55:23.4150222Z deleted: sha256:b4ee4f97dcc06aa686efba9fa49bb8ed5a5f9ee5e555dfd1854c4f617ccdde42 2023-11-14T07:55:23.4151304Z deleted: sha256:9c80008875379b87b4eb54e0a202471743e15020597b6ce1081a47ce3e243657 2023-11-14T07:55:23.4152335Z deleted: sha256:478b4d9242bf51f25afaee8a17668e62354eaf0f037498ef7974207f09dd9fc0 2023-11-14T07:55:23.4153381Z deleted: sha256:fd3dce86797122ae34ab925851b8018328f36db5add7ce0a6c59ec42782a9253 2023-11-14T07:55:23.4154432Z deleted: sha256:07d43fc41ab24cf9a60d85cedc219939a6c0e13e5da54b2b3b1ba04ae335f8a0 2023-11-14T07:55:23.4155514Z deleted: sha256:3b0bf5a23b0af98b94fde5c294d6cdb7179f52d5bfaea66c96fcdef51bc17494 2023-11-14T07:55:23.4156595Z deleted: sha256:7b22f709c4f3b882fe0f7b19c7d2722f1beffe00fdc21601d8725b2ec758cb2d 2023-11-14T07:55:23.4157649Z deleted: sha256:0398ace2cfc2753c45b24e112fd3e2ed823c2a3cbf69bf70c7ea7bf5d90a4d5d 2023-11-14T07:55:23.4158713Z deleted: sha256:91a560086e3b6ef014ab3c6b1f20bfa2a33267f4e338a54566c717c11542966b 2023-11-14T07:55:23.4159797Z deleted: sha256:c8300e5affc1abe3fde52a5fbdb868af96a8c8ac4293ea5fa0861dea94b086bf 2023-11-14T07:55:23.4160878Z deleted: sha256:6fe25eaec572795438d50d127b23a949366874beec0c20e82b69f6909b09a3a4 2023-11-14T07:55:23.4161924Z deleted: sha256:b676deccd8574b11ab6baeabc60868f51d5745f73a5df8293ae6caccd82d5757 2023-11-14T07:55:23.4163254Z deleted: sha256:7f0fa62b771f33c2743916a00800c0b7ef42397b25107e6bfd5d2ae7d05d3661 2023-11-14T07:55:23.4164287Z deleted: sha256:97b766680dda9399a9703808320b8142ab92653b8a66b7356a81818e031b1c31 2023-11-14T07:55:23.4165302Z deleted: sha256:a63c13b07c37c9db4762fb6ded0a310801775ab21694d7357f5adc547589f152 2023-11-14T07:55:23.4166354Z deleted: sha256:716f2784f05e703b3e8e997219994fff81f33da47346da6fe92b3b5dd52afe90 2023-11-14T07:55:23.4167393Z deleted: sha256:218d7338e785bbec77a628e85f5c0042d627ba7ef2785407740f960744e9c031 2023-11-14T07:55:23.4168403Z deleted: sha256:fc92863fa41959f0178a6cc629583b23677afcf0179eb394ad4301345ad86b7a 2023-11-14T07:55:23.4169902Z deleted: sha256:28efec4552900781978c637546ebf0199d26c1e56ee4e17fe392059c932ddac2 2023-11-14T07:55:23.4171626Z deleted: sha256:3bdcccc186266e4aeb82fe136daf065d3edee2ad2d0f49e29aa0bf2aa0d8f343 2023-11-14T07:55:23.4172772Z deleted: sha256:54e5f78fa13bc8fabfeb36f48b88e7a84380a00d12ccbc184a16513a115076a9 2023-11-14T07:55:23.4173807Z deleted: sha256:59e468e5df4e579b108d9298d26dca1701012533337f3448f04156a24c6cd076 2023-11-14T07:55:23.4174868Z deleted: sha256:c8debf67eb97b09bc5582acd540435d55afbcfe3e074f5a3c117e37db387de29 2023-11-14T07:55:23.4175937Z deleted: sha256:4ef0affb5a7229892f3928318556d8d9b688f567daf1da4225b0e0950cba8fcb 2023-11-14T07:55:23.4176957Z deleted: sha256:902999e13a0732020040c9fec1b3d5f731bf15c96b759b3a81053afc0f421840 2023-11-14T07:55:23.4178005Z deleted: sha256:70467a5ca7e57d18e5748ae93aa7050fe8860d0da0a558b21d3deafcc2d0b636 2023-11-14T07:55:23.4179051Z deleted: sha256:90479113476838afd72a83b5b42cdf6e75574f00af3f820fe34977b5f3fd2548 2023-11-14T07:55:23.4180095Z deleted: sha256:ab01c4657ada85c5bff93428e0493e9bffb26fe7604cf1c84919ff2c0289029f 2023-11-14T07:55:23.4181189Z deleted: sha256:864065ffcddbc3e8d476f5bf00831656acefbdf4e0e7ddfa5deddbcea116ab74 2023-11-14T07:55:23.4182261Z deleted: sha256:5f277e57fd48ab88775573de37d122ca48dae81f4e8d988ca27b876523f76598 2023-11-14T07:55:23.4183310Z deleted: sha256:606f9756a15c988a2bdda76a0d691866d94632ad66845486b599d33ebf3f1960 2023-11-14T07:55:23.4184343Z deleted: sha256:1a185cc315e63cd72c51b03ef6d959c8f6ad0e31b165fe28deefb04e81e23fbf 2023-11-14T07:55:23.4185467Z deleted: sha256:11106898dfd6aa5a82a26bb1adfa3b96cfd17b46004e6f798408b8a4ff90a20e 2023-11-14T07:55:23.4186527Z deleted: sha256:d5b52f1c2127024e85dec1b5313c886e9856efaf4beb5d1625861b0a45417784 2023-11-14T07:55:23.4187546Z deleted: sha256:350f24064adcc53c861785ed453c62fa78c810eb6272144307367af2237523f8 2023-11-14T07:55:23.4188602Z deleted: sha256:d5e1eb7e991eb872cbeda851811d29f073f51604aa7aeaeaeb11a404be874b83 2023-11-14T07:55:23.4189661Z deleted: sha256:a0f524642395fd3247e6fe0f53b91a520b1a6b132521ed9ab5d7481a867707e6 2023-11-14T07:55:23.4190693Z deleted: sha256:b7d7a9df07683914a58a4a4415b7c341536391af96904c6ae7f5c1f31dc1ee7e 2023-11-14T07:55:23.4191730Z deleted: sha256:37deeab68757221d4740d97062f42d85962ba2a179e895b0234d73681c140219 2023-11-14T07:55:23.4192769Z deleted: sha256:347b7d34ff056fa73466bd5774158bd466e7392c2b95c5cb6ed383c0344296f3 2023-11-14T07:55:23.4193832Z deleted: sha256:ec66d8cea54a2f4dfbbd8342ce082503bf8541e996a800c0d724b8dd2fea7f6a 2023-11-14T07:55:23.4194483Z 2023-11-14T07:55:23.4194658Z Total reclaimed space: 25.61GB 2023-11-14T07:55:23.4259732Z ##[group]Run set +e 2023-11-14T07:55:23.4260267Z set +e 2023-11-14T07:55:23.4260629Z set -x 2023-11-14T07:55:23.4260965Z  2023-11-14T07:55:23.4261297Z nvidia-smi 2023-11-14T07:55:23.4262074Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2023-11-14T07:55:23.4263302Z # the case where the driver has already crashed as it still can get the driver version 2023-11-14T07:55:23.4264538Z # and some basic information like the bus ID. However, the rest of the information 2023-11-14T07:55:23.4265467Z # would be missing (ERR!), for example: 2023-11-14T07:55:23.4266001Z # 2023-11-14T07:55:23.4266516Z # +-----------------------------------------------------------------------------+ 2023-11-14T07:55:23.4267476Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2023-11-14T07:55:23.4268468Z # |-------------------------------+----------------------+----------------------+ 2023-11-14T07:55:23.4269423Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2023-11-14T07:55:23.4270816Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2023-11-14T07:55:23.4271743Z # | | | MIG M. | 2023-11-14T07:55:23.4272666Z # |===============================+======================+======================| 2023-11-14T07:55:23.4273471Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2023-11-14T07:55:23.4274389Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2023-11-14T07:55:23.4275263Z # | | | ERR! | 2023-11-14T07:55:23.4276052Z # +-------------------------------+----------------------+----------------------+ 2023-11-14T07:55:23.4276677Z # 2023-11-14T07:55:23.4277181Z # +-----------------------------------------------------------------------------+ 2023-11-14T07:55:23.4278015Z # | Processes: | 2023-11-14T07:55:23.4278931Z # | GPU GI CI PID Type Process name GPU Memory | 2023-11-14T07:55:23.4279812Z # | ID ID Usage | 2023-11-14T07:55:23.4280568Z # |=============================================================================| 2023-11-14T07:55:23.4281338Z # +-----------------------------------------------------------------------------+ 2023-11-14T07:55:23.4281958Z # 2023-11-14T07:55:23.4282638Z # This should be reported as a failure instead as it will guarantee to fail when 2023-11-14T07:55:23.4283541Z # Docker tries to run with --gpus all 2023-11-14T07:55:23.4284085Z # 2023-11-14T07:55:23.4284723Z # So, the correct check here is to query one of the missing piece of info like 2023-11-14T07:55:23.4285762Z # GPU name, so that the command can fail accordingly 2023-11-14T07:55:23.4286625Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2023-11-14T07:55:23.4287323Z NVIDIA_SMI_STATUS=$? 2023-11-14T07:55:23.4287777Z  2023-11-14T07:55:23.4288536Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2023-11-14T07:55:23.4289675Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2023-11-14T07:55:23.4291189Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2023-11-14T07:55:23.4292173Z  .github/scripts/stop_runner_service.sh 2023-11-14T07:55:23.4292739Z fi 2023-11-14T07:55:23.4293054Z  2023-11-14T07:55:23.4293932Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2023-11-14T07:55:23.4295169Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2023-11-14T07:55:23.4296153Z # https://github.com/pytorch/test-infra/issues/4000 2023-11-14T07:55:23.4296899Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2023-11-14T07:55:23.4297505Z NVIDIA_SMI_STATUS=$? 2023-11-14T07:55:23.4297931Z  2023-11-14T07:55:23.4298693Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2023-11-14T07:55:23.4299845Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2023-11-14T07:55:23.4300881Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2023-11-14T07:55:23.4301754Z  .github/scripts/stop_runner_service.sh 2023-11-14T07:55:23.4302317Z fi 2023-11-14T07:55:23.4302659Z  2023-11-14T07:55:23.4303071Z # Check the GPU count to be a power of 2 2023-11-14T07:55:23.4304123Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2023-11-14T07:55:23.4305512Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2023-11-14T07:55:23.4307086Z  .github/scripts/stop_runner_service.sh 2023-11-14T07:55:23.4307632Z fi 2023-11-14T07:55:23.4316198Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:55:23.4316821Z env: 2023-11-14T07:55:23.4317159Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:55:23.4317700Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:55:23.4318628Z DOCKER_CONTAINER_ID: 99f90b6e6ff3e9a68f028af62c75c330cdf5b58c22500959645fa66d471129fe 2023-11-14T07:55:23.4319607Z RUNNER_WORKSPACE: /home/ec2-user/actions-runner/_work/pytorch 2023-11-14T07:55:23.4320243Z ##[endgroup] 2023-11-14T07:55:23.4349160Z + nvidia-smi 2023-11-14T07:55:24.7958813Z Tue Nov 14 07:55:24 2023 2023-11-14T07:55:24.7959690Z +---------------------------------------------------------------------------------------+ 2023-11-14T07:55:24.7960721Z | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2023-11-14T07:55:24.7961702Z |-----------------------------------------+----------------------+----------------------+ 2023-11-14T07:55:24.7962719Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2023-11-14T07:55:24.7963808Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2023-11-14T07:55:24.7964670Z | | | MIG M. | 2023-11-14T07:55:24.7965344Z |=========================================+======================+======================| 2023-11-14T07:55:24.8006066Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2023-11-14T07:55:24.8006963Z | N/A 26C P0 39W / 150W | 0MiB / 7680MiB | 83% Default | 2023-11-14T07:55:24.8007758Z | | | N/A | 2023-11-14T07:55:24.8008592Z +-----------------------------------------+----------------------+----------------------+ 2023-11-14T07:55:24.8054568Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2023-11-14T07:55:24.8055467Z | N/A 28C P0 38W / 150W | 0MiB / 7680MiB | 9% Default | 2023-11-14T07:55:24.8056255Z | | | N/A | 2023-11-14T07:55:24.8057064Z +-----------------------------------------+----------------------+----------------------+ 2023-11-14T07:55:24.8057777Z 2023-11-14T07:55:24.8058866Z +---------------------------------------------------------------------------------------+ 2023-11-14T07:55:24.8059692Z | Processes: | 2023-11-14T07:55:24.8060550Z | GPU GI CI PID Type Process name GPU Memory | 2023-11-14T07:55:24.8061424Z | ID ID Usage | 2023-11-14T07:55:24.8062131Z |=======================================================================================| 2023-11-14T07:55:24.8062922Z | No running processes found | 2023-11-14T07:55:24.8063872Z +---------------------------------------------------------------------------------------+ 2023-11-14T07:55:25.7487970Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2023-11-14T07:55:27.1395862Z Tesla M60 2023-11-14T07:55:28.0125457Z + NVIDIA_SMI_STATUS=0 2023-11-14T07:55:28.0132266Z + '[' 0 -ne 0 ']' 2023-11-14T07:55:28.0132732Z ++ nvidia-smi --list-gpus 2023-11-14T07:55:28.0133165Z ++ wc -l 2023-11-14T07:55:30.2885865Z + GPU_COUNT=2 2023-11-14T07:55:30.2886326Z + NVIDIA_SMI_STATUS=0 2023-11-14T07:55:30.2886881Z + '[' 0 -ne 0 ']' 2023-11-14T07:55:30.2887302Z + '[' 2 -le 8 ']' 2023-11-14T07:55:30.2887707Z + '[' 2 -ne 1 ']' 2023-11-14T07:55:30.2888130Z + '[' 2 -ne 2 ']' 2023-11-14T07:55:30.2965254Z Post job cleanup. 2023-11-14T07:55:30.3032175Z Post job cleanup. 2023-11-14T07:55:30.4204396Z [command]/usr/bin/git version 2023-11-14T07:55:30.4258070Z git version 2.40.1 2023-11-14T07:55:30.4322314Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a64236d7-61cc-4a06-a6da-e01e22df37ae' before making global git config changes 2023-11-14T07:55:30.4323917Z Adding repository directory to the temporary git global config as a safe directory 2023-11-14T07:55:30.4331162Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T07:55:30.4373834Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2023-11-14T07:55:30.4411923Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2023-11-14T07:55:30.4740674Z Entering 'android/libs/fbjni' 2023-11-14T07:55:30.4795695Z Entering 'third_party/FP16' 2023-11-14T07:55:30.4850901Z Entering 'third_party/FXdiv' 2023-11-14T07:55:30.4909242Z Entering 'third_party/NNPACK' 2023-11-14T07:55:30.4964394Z Entering 'third_party/QNNPACK' 2023-11-14T07:55:30.5020309Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T07:55:30.5075115Z Entering 'third_party/XNNPACK' 2023-11-14T07:55:30.5145767Z Entering 'third_party/benchmark' 2023-11-14T07:55:30.5200453Z Entering 'third_party/cpuinfo' 2023-11-14T07:55:30.5256934Z Entering 'third_party/cub' 2023-11-14T07:55:30.5312869Z Entering 'third_party/cudnn_frontend' 2023-11-14T07:55:30.5375636Z Entering 'third_party/cutlass' 2023-11-14T07:55:30.5441674Z Entering 'third_party/eigen' 2023-11-14T07:55:30.5500817Z Entering 'third_party/fbgemm' 2023-11-14T07:55:30.5556091Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T07:55:30.5611710Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T07:55:30.5667619Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T07:55:30.5731546Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T07:55:30.5787156Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T07:55:30.5844067Z Entering 'third_party/flatbuffers' 2023-11-14T07:55:30.5903842Z Entering 'third_party/fmt' 2023-11-14T07:55:30.5959379Z Entering 'third_party/foxi' 2023-11-14T07:55:30.6016259Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T07:55:30.6071542Z Entering 'third_party/gloo' 2023-11-14T07:55:30.6127131Z Entering 'third_party/googletest' 2023-11-14T07:55:30.6185137Z Entering 'third_party/ideep' 2023-11-14T07:55:30.6239279Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T07:55:30.6301067Z Entering 'third_party/ios-cmake' 2023-11-14T07:55:30.6356233Z Entering 'third_party/ittapi' 2023-11-14T07:55:30.6412436Z Entering 'third_party/kineto' 2023-11-14T07:55:30.6467088Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T07:55:30.6523441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T07:55:30.6582457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T07:55:30.6637283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T07:55:30.6692845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T07:55:30.6748947Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T07:55:30.6806513Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T07:55:30.6862609Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T07:55:30.6919419Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T07:55:30.6977916Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T07:55:30.7034502Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T07:55:30.7092224Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T07:55:30.7148675Z Entering 'third_party/mimalloc' 2023-11-14T07:55:30.7206431Z Entering 'third_party/nccl/nccl' 2023-11-14T07:55:30.7263568Z Entering 'third_party/neon2sse' 2023-11-14T07:55:30.7318477Z Entering 'third_party/nlohmann' 2023-11-14T07:55:30.7376879Z Entering 'third_party/onnx' 2023-11-14T07:55:30.7451011Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T07:55:30.7506761Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T07:55:30.7564888Z Entering 'third_party/onnx-tensorrt' 2023-11-14T07:55:30.7621471Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T07:55:30.7681778Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T07:55:30.7739007Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T07:55:30.7793939Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T07:55:30.7854431Z Entering 'third_party/pocketfft' 2023-11-14T07:55:30.7910971Z Entering 'third_party/protobuf' 2023-11-14T07:55:30.7970418Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T07:55:30.8025247Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T07:55:30.8083310Z Entering 'third_party/psimd' 2023-11-14T07:55:30.8141548Z Entering 'third_party/pthreadpool' 2023-11-14T07:55:30.8197677Z Entering 'third_party/pybind11' 2023-11-14T07:55:30.8254239Z Entering 'third_party/python-peachpy' 2023-11-14T07:55:30.8309872Z Entering 'third_party/sleef' 2023-11-14T07:55:30.8365419Z Entering 'third_party/tbb' 2023-11-14T07:55:30.8425944Z Entering 'third_party/tensorpipe' 2023-11-14T07:55:30.8481683Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T07:55:30.8536979Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T07:55:30.8592166Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T07:55:30.8646929Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T07:55:30.8701405Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T07:55:30.8758949Z Entering 'third_party/zstd' 2023-11-14T07:55:30.8833989Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2023-11-14T07:55:30.8862032Z http.https://github.com/.extraheader 2023-11-14T07:55:30.8873865Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2023-11-14T07:55:30.8911665Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2023-11-14T07:55:30.9229097Z Entering 'android/libs/fbjni' 2023-11-14T07:55:30.9266198Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9301599Z Entering 'third_party/FP16' 2023-11-14T07:55:30.9339563Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9374204Z Entering 'third_party/FXdiv' 2023-11-14T07:55:30.9411136Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9445437Z Entering 'third_party/NNPACK' 2023-11-14T07:55:30.9481821Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9517361Z Entering 'third_party/QNNPACK' 2023-11-14T07:55:30.9554262Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9589164Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T07:55:30.9626753Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9663035Z Entering 'third_party/XNNPACK' 2023-11-14T07:55:30.9700324Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9750311Z Entering 'third_party/benchmark' 2023-11-14T07:55:30.9787238Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9822823Z Entering 'third_party/cpuinfo' 2023-11-14T07:55:30.9859175Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9894372Z Entering 'third_party/cub' 2023-11-14T07:55:30.9931512Z http.https://github.com/.extraheader 2023-11-14T07:55:30.9967381Z Entering 'third_party/cudnn_frontend' 2023-11-14T07:55:31.0004830Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0045749Z Entering 'third_party/cutlass' 2023-11-14T07:55:31.0082022Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0123582Z Entering 'third_party/eigen' 2023-11-14T07:55:31.0160044Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0198429Z Entering 'third_party/fbgemm' 2023-11-14T07:55:31.0234971Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0270776Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T07:55:31.0306707Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0343133Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T07:55:31.0379041Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0414797Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T07:55:31.0450099Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0492953Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T07:55:31.0529205Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0564050Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T07:55:31.0599854Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0636214Z Entering 'third_party/flatbuffers' 2023-11-14T07:55:31.0673058Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0710258Z Entering 'third_party/fmt' 2023-11-14T07:55:31.0746378Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0780787Z Entering 'third_party/foxi' 2023-11-14T07:55:31.0818046Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0851931Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T07:55:31.0888167Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0922797Z Entering 'third_party/gloo' 2023-11-14T07:55:31.0959209Z http.https://github.com/.extraheader 2023-11-14T07:55:31.0993951Z Entering 'third_party/googletest' 2023-11-14T07:55:31.1030711Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1064993Z Entering 'third_party/ideep' 2023-11-14T07:55:31.1101076Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1136107Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T07:55:31.1171225Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1213270Z Entering 'third_party/ios-cmake' 2023-11-14T07:55:31.1249361Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1283591Z Entering 'third_party/ittapi' 2023-11-14T07:55:31.1321030Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1355676Z Entering 'third_party/kineto' 2023-11-14T07:55:31.1392556Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1427614Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T07:55:31.1463174Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1500085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T07:55:31.1536869Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1574874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T07:55:31.1610433Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1646792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T07:55:31.1683327Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1718473Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T07:55:31.1755095Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1789799Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T07:55:31.1826585Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1864011Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T07:55:31.1900414Z http.https://github.com/.extraheader 2023-11-14T07:55:31.1935975Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T07:55:31.1971766Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2007747Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T07:55:31.2045289Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2082293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T07:55:31.2118371Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2155672Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T07:55:31.2191714Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2227647Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T07:55:31.2264052Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2300501Z Entering 'third_party/mimalloc' 2023-11-14T07:55:31.2337539Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2372517Z Entering 'third_party/nccl/nccl' 2023-11-14T07:55:31.2409858Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2444754Z Entering 'third_party/neon2sse' 2023-11-14T07:55:31.2481309Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2515788Z Entering 'third_party/nlohmann' 2023-11-14T07:55:31.2552139Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2587486Z Entering 'third_party/onnx' 2023-11-14T07:55:31.2624050Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2676968Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T07:55:31.2713534Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2749088Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T07:55:31.2785127Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2822744Z Entering 'third_party/onnx-tensorrt' 2023-11-14T07:55:31.2859913Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2894676Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T07:55:31.2929534Z http.https://github.com/.extraheader 2023-11-14T07:55:31.2971129Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T07:55:31.3006658Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3042958Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T07:55:31.3079375Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3115162Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T07:55:31.3151510Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3191512Z Entering 'third_party/pocketfft' 2023-11-14T07:55:31.3228039Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3263115Z Entering 'third_party/protobuf' 2023-11-14T07:55:31.3300478Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3339666Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T07:55:31.3376527Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3411327Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T07:55:31.3446982Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3484412Z Entering 'third_party/psimd' 2023-11-14T07:55:31.3521768Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3556493Z Entering 'third_party/pthreadpool' 2023-11-14T07:55:31.3594439Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3629463Z Entering 'third_party/pybind11' 2023-11-14T07:55:31.3665748Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3701517Z Entering 'third_party/python-peachpy' 2023-11-14T07:55:31.3738829Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3773454Z Entering 'third_party/sleef' 2023-11-14T07:55:31.3809921Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3845205Z Entering 'third_party/tbb' 2023-11-14T07:55:31.3882635Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3919787Z Entering 'third_party/tensorpipe' 2023-11-14T07:55:31.3956352Z http.https://github.com/.extraheader 2023-11-14T07:55:31.3991274Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T07:55:31.4027226Z http.https://github.com/.extraheader 2023-11-14T07:55:31.4062698Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T07:55:31.4099159Z http.https://github.com/.extraheader 2023-11-14T07:55:31.4133644Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T07:55:31.4168807Z http.https://github.com/.extraheader 2023-11-14T07:55:31.4204715Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T07:55:31.4240483Z http.https://github.com/.extraheader 2023-11-14T07:55:31.4275012Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T07:55:31.4311353Z http.https://github.com/.extraheader 2023-11-14T07:55:31.4350235Z Entering 'third_party/zstd' 2023-11-14T07:55:31.4386641Z http.https://github.com/.extraheader 2023-11-14T07:55:31.4738923Z Cleaning up orphan processes